See-through near-eye display glasses including an auto-brightness control for the display brightness based on the brightness in the environment

ABSTRACT

This disclosure concerns an interactive head-mounted eyepiece with an integrated processor for handling content for display and an integrated image source for introducing the content to an optical assembly through which the user views a surrounding environment and the displayed content. The image source includes an auto-brightness control that controls the brightness of the displayed content based on the brightness in the environment.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application61/584,029, filed Jan. 6, 2012, which is incorporated herein byreference in its entirety.

This application is a continuation-in-part of the following UnitedStates non-provisional patent applications, each of which isincorporated herein by reference in its entirety:

U.S. Non-Provisional application Ser. No. 13/341,758, filed Dec. 30,2011, which claims the benefit of the following provisionalapplications, each of which is hereby incorporated herein by referencein its entirety: U.S. Provisional Patent Application 61/557,289, filedNov. 8, 2011.

U.S. Non-Provisional application Ser. No. 13/232,930, filed Sep. 14,2011, which claims the benefit of the following provisionalapplications, each of which is hereby incorporated herein by referencein its entirety: U.S. Provisional Application 61/382,578, filed Sep. 14,2010; U.S. Provisional Application 61/472,491, filed Apr. 6, 2011; U.S.Provisional Application 61/483,400, filed May 6, 2011; U.S. ProvisionalApplication 61/487,371, filed May 18, 2011; and U.S. ProvisionalApplication 61/504,513, filed Jul. 5, 2011.

U.S. patent application Ser. No. 13/037,324, filed Feb. 28, 2011 andU.S. patent application Ser. No. 13/037,335, filed Feb. 28, 2011, eachof which claim the benefit of the following provisional applications,each of which is hereby incorporated herein by reference in itsentirety: U.S. Provisional Patent Application 61/308,973, filed Feb. 28,2010; U.S. Provisional Patent Application 61/373,791, filed Aug. 13,2010; U.S. Provisional Patent Application 61/382,578, filed Sep. 14,2010; U.S. Provisional Patent Application 61/410,983, filed Nov. 8,2010; U.S. Provisional Patent Application 61/429,445, filed Jan. 3,2011; and U.S. Provisional Patent Application 61/429,447, filed Jan. 3,2011.

BACKGROUND Field

The present disclosure relates to an augmented reality eyepiece,associated control technologies, and applications for use, and morespecifically to software applications running on the eyepiece.

SUMMARY

In embodiments, the eyepiece may include an internal softwareapplication running on an integrated multimedia computing facility thathas been adapted for 3D augmented reality (AR) content display andinteraction with the eyepiece. 3D AR software applications may bedeveloped in conjunction with mobile applications and provided throughapplication store(s), or as stand-alone applications specificallytargeting the eyepiece as the end-use platform and through a dedicated3D AR eyepiece store. Internal software applications may interface withinputs and output facilities provided by the eyepiece through facilitiesinternal and external to the eyepiece, such as initiated from thesurrounding environment, sensing devices, user action capture devices,internal processing facilities, internal multimedia processingfacilities, other internal applications, camera, sensors, microphone,through a transceiver, through a tactile interface, from externalcomputing facilities, external applications, event and/or data feeds,external devices, third parties, and the like. Command and control modesoperating in conjunction with the eyepiece may be initiated by sensinginputs through input devices, user action, external device interaction,reception of events and/or data feeds, internal application execution,external application execution, and the like. In embodiments, there maybe a series of steps included in the execution control as providedthrough the internal software application, including at leastcombinations of two of the following: events and/or data feeds, sensinginputs and/or sensing devices, user action capture inputs and/oroutputs, user movements and/or actions for controlling and/or initiatingcommands, command and/or control modes and interfaces in which theinputs may be reflected, applications on the platform that may usecommands to respond to inputs, communications and/or connection from theon-platform interface to external systems and/or devices, externaldevices, external applications, feedback to the user (such as related toexternal devices, external applications), and the like.

These and other systems, methods, objects, features, and advantages ofthe present disclosure will be apparent to those skilled in the art fromthe following detailed description of the embodiments and the drawings.

All documents mentioned herein are hereby incorporated in their entiretyby reference. References to items in the singular should be understoodto include items in the plural, and vice versa, unless explicitly statedotherwise or clear from the text. Grammatical conjunctions are intendedto express any and all disjunctive and conjunctive combinations ofconjoined clauses, sentences, words, and the like, unless otherwisestated or clear from the context.

BRIEF DESCRIPTION OF THE FIGURES

The present disclosure and the following detailed description of certainembodiments thereof may be understood by reference to the followingfigures:

FIG. 1 depicts an illustrative embodiment of the optical arrangement.

FIG. 2 depicts an RGB LED projector.

FIG. 3 depicts the projector in use.

FIG. 4 depicts an embodiment of the waveguide and correction lensdisposed in a frame.

FIG. 5 depicts a design for a waveguide eyepiece.

FIG. 6 depicts an embodiment of the eyepiece with a see-through lens.

FIG. 7 depicts an embodiment of the eyepiece with a see-through lens.

FIG. 8A-C depicts embodiments of the eyepiece arranged in aflip-up/flip-down configuration.

FIG. 8D-E depicts embodiments of snap-fit elements of a secondary optic.

FIG. 8F depicts embodiments of flip-up/flip-down electro-optics modules.

FIG. 9 depicts an electrochromic layer of the eyepiece.

FIG. 10 depicts the advantages of the eyepiece in real-time imageenhancement, keystone correction, and virtual perspective correction.

FIG. 11 depicts a plot of responsivity versus wavelength for threesubstrates.

FIG. 12 illustrates the performance of the black silicon sensor.

FIG. 13A depicts an incumbent night vision system, FIG. 13B depicts thenight vision system of the present disclosure, and FIG. 13C illustratesthe difference in responsivity between the two.

FIG. 14 depicts a tactile interface of the eyepiece.

FIG. 14A depicts motions in an embodiment of the eyepiece featuring nodcontrol.

FIG. 15 depicts a ring that controls the eyepiece.

FIG. 15AA depicts a ring that controls the eyepiece with an integratedcamera, where in an embodiment may allow the user to provide a videoimage of themselves as part of a videoconference.

FIG. 15A depicts hand mounted sensors in an embodiment of a virtualmouse.

FIG. 15B depicts a facial actuation sensor as mounted on the eyepiece.

FIG. 15C depicts a hand pointing control of the eyepiece.

FIG. 15D depicts a hand pointing control of the eyepiece.

FIG. 15E depicts an example of eye tracking control.

FIG. 15F depicts a hand positioning control of the eyepiece.

FIG. 16 depicts a location-based application mode of the eyepiece.

FIG. 17 shows the difference in image quality between A) a flexibleplatform of uncooled CMOS image sensors capable of VIS/NIR/SWIR imagingand B) an image intensified night vision system

FIG. 18 depicts an augmented reality-enabled custom billboard.

FIG. 19 depicts an augmented reality-enabled custom advertisement.

FIG. 20 an augmented reality-enabled custom artwork.

FIG. 20A depicts a method for posting messages to be transmitted when aviewer reaches a certain location.

FIG. 21 depicts an alternative arrangement of the eyepiece optics andelectronics.

FIG. 22 depicts an alternative arrangement of the eyepiece optics andelectronics.

FIG. 22A depicts the eyepiece with an example of eyeglow.

FIG. 22B depicts a cross-section of the eyepiece with a light controlelement for reducing eyeglow.

FIG. 23 depicts an alternative arrangement of the eyepiece optics andelectronics.

FIG. 24 depicts a lock position of a virtual keyboard.

FIG. 24A depicts an embodiment of a virtually projected image on a partof the human body.

FIG. 25 depicts a detailed view of the projector.

FIG. 26 depicts a detailed view of the RGB LED module.

FIG. 27 depicts a gaming network.

FIG. 28 depicts a method for gaming using augmented reality glasses.

FIG. 29 depicts an exemplary electronic circuit diagram for an augmentedreality eyepiece.

FIG. 29A depicts a control circuit for eye-tracking control of anexternal device.

FIG. 29B depicts a communication network among users of augmentedreality eyepieces.

FIG. 30 depicts partial image removal by the eyepiece.

FIG. 31 depicts a flowchart for a method of identifying a person basedon speech of the person as captured by microphones of the augmentedreality device.

FIG. 32 depicts a typical camera for use in video calling orconferencing.

FIG. 33 illustrates an embodiment of a block diagram of a video callingcamera.

FIG. 34 depicts embodiments of the eyepiece for optical or digitalstabilization.

FIG. 35 depicts an embodiment of a classic cassegrain configuration.

FIG. 36 depicts the configuration of the micro-cassegrain telescopingfolded optic camera.

FIG. 37 depicts a swipe process with a virtual keyboard.

FIG. 38 depicts a target marker process for a virtual keyboard.

FIG. 38A depicts an embodiment of a visual word translator.

FIG. 39 illustrates glasses for biometric data capture according to anembodiment.

FIG. 40 illustrates iris recognition using the biometric data captureglasses according to an embodiment.

FIG. 41 depicts face and iris recognition according to an embodiment.

FIG. 42 illustrates use of dual omni-microphones according to anembodiment.

FIG. 43 depicts the directionality improvements with multiplemicrophones.

FIG. 44 shows the use of adaptive arrays to steer the audio capturefacility according to an embodiment.

FIG. 45 shows the mosaic finger and palm enrollment system according toan embodiment.

FIG. 46 illustrates the traditional optical approach used by otherfinger and palm print systems.

FIG. 47 shows the approach used by the mosaic sensor according to anembodiment.

FIG. 48 depicts the device layout of the mosaic sensor according to anembodiment.

FIG. 49 illustrates the camera field of view and number of cameras usedin a mosaic sensor according to another embodiment.

FIG. 50 shows the bio-phone and tactical computer according to anembodiment.

FIG. 51 shows the use of the bio-phone and tactical computer incapturing latent fingerprints and palm prints according to anembodiment.

FIG. 52 illustrates a typical DOMEX collection.

FIG. 53 shows the relationship between the biometric images capturedusing the bio-phone and tactical computer and a biometric watch listaccording to an embodiment.

FIG. 54 illustrates a pocket bio-kit according to an embodiment.

FIG. 55 shows the components of the pocket bio-kit according to anembodiment.

FIG. 56 depicts the fingerprint, palm print, geo-location and POIenrollment device according to an embodiment.

FIG. 57 shows a system for multi-modal biometric collection,identification, geo-location, and POI enrollment according to anembodiment.

FIG. 58 illustrates a fingerprint, palm print, geo-location, and POIenrollment forearm wearable device according to an embodiment.

FIG. 59 shows a mobile folding biometric enrollment kit according to anembodiment.

FIG. 60 is a high level system diagram of a biometric enrollment kitaccording to an embodiment.

FIG. 61 is a system diagram of a folding biometric enrollment deviceaccording to an embodiment.

FIG. 62 shows a thin-film finger and palm print sensor according to anembodiment.

FIG. 63 shows a biometric collection device for finger, palm, andenrollment data collection according to an embodiment.

FIG. 64 illustrates capture of a two stage palm print according to anembodiment.

FIG. 65 illustrates capture of a fingertip tap according to anembodiment.

FIG. 66 illustrates capture of a slap and roll print according to anembodiment.

FIG. 67 depicts a system for taking contactless fingerprints, palmprintsor other biometric prints.

FIG. 68 depicts a process for taking contactless fingerprints,palmprints or other biometric prints.

FIG. 69 depicts an embodiment of a watch controller.

FIG. 70A-D depicts embodiment cases for the eyepiece, includingcapabilities for charging and integrated display.

FIG. 71 depicts an embodiment of a ground stake data system.

FIG. 72 depicts a block diagram of a control mapping system includingthe eyepiece.

FIG. 73 depicts a biometric flashlight.

FIG. 74 depicts a helmet-mounted version of the eyepiece.

FIG. 75 depicts an embodiment of situational awareness glasses.

FIG. 76A depicts an assembled 360° imager and FIG. 76B depicts a cutawayview of the 360° imager.

FIG. 77 depicts an exploded view of the multi-coincident view camera.

FIG. 78 depicts a flight eye.

FIG. 79 depicts an exploded top view of the eyepiece.

FIG. 80 depicts an exploded electro-optic assembly.

FIG. 81 depicts an exploded view of the shaft of the electro-opticassembly.

FIG. 82 depicts an embodiment of an optical display system utilizing aplanar illumination facility with a reflective display.

FIG. 83 depicts a structural embodiment of a planar illumination opticalsystem.

FIG. 84 depicts an embodiment assembly of a planar illumination facilityand a reflective display with laser speckle suppression components.

FIG. 85 depicts an embodiment of a planar illumination facility withgrooved features for redirecting light.

FIG. 86 depicts an embodiment of a planar illumination facility withgrooved features and ‘anti-grooved’ features paired to reduce imageaberrations.

FIG. 87 depicts an embodiment of a planar illumination facilityfabricated from a laminate structure.

FIG. 88 depicts an embodiment of a planar illumination facility with awedged optic assembly for redirecting light.

FIG. 89 depicts a block diagram of an illumination module, according toan embodiment of the invention.

FIG. 90 depicts a block diagram of an optical frequency converter,according to an embodiment of the invention.

FIG. 91 depicts a block diagram of a laser illumination module,according to an embodiment of the invention.

FIG. 92 depicts a block diagram of a laser illumination system,according to another embodiment of the invention.

FIG. 93 depicts a block diagram of an imaging system, according to anembodiment of the invention.

FIGS. 94A & B depict a lens with a photochromic element and a heaterelement in a top down and side view, respectively.

FIG. 95 depicts an embodiment of an LCoS front light design.

FIG. 96 depicts optically bonded prisms with a polarizer.

FIG. 97 depicts optically bonded prisms with a polarizer.

FIG. 98 depicts multiple embodiments of an LCoS front light design.

FIG. 99 depicts a wedge plus OBS overlaid on an LCoS.

FIG. 100 depicts two versions of a wedge.

FIG. 101 depicts a curved PBS film over the LCoS chip.

FIG. 102 depicts an embodiment of an optical assembly.

FIG. 103 depicts an embodiment of an image source.

FIG. 104 depicts an embodiment of an image source.

FIG. 105 depicts embodiments of image sources.

FIG. 106 depicts a top-level block diagram showing software applicationfacilities and markets in conjunction with functional and controlaspects of the eyepiece in an embodiment of the present invention.

FIG. 107 depicts a functional block diagram of the eyepiece applicationdevelopment environment in an embodiment of the present invention.

FIG. 108 depicts a platform elements development stack in relation tosoftware applications for the eyepiece in an embodiment of the presentinvention.

FIG. 109 is an illustration of a head mounted display with see-throughcapability according to an embodiment of the present invention.

FIG. 110 is an illustration of a view of an unlabeled scene as viewedthrough the head mounted display depicted in FIG. 109.

FIG. 111 is an illustration of a view of the scene of FIG. 110 with 2Doverlaid labels.

FIG. 112 is an illustration of 3D labels of FIG. 111 as displayed to theviewer's left eye.

FIG. 113 is an illustration of 3D labels of FIG. 111 as displayed to theviewer's right eye.

FIG. 114 is an illustration of the left and right 3D labels of FIG. 111overlaid on one another to show the disparity.

FIG. 115 is an illustration of the view of a scene of FIG. 110 with the3D labels.

FIG. 116 is an illustration of stereo images captured of the scene ofFIG. 110.

FIG. 117 is an illustration of the overlaid left and right stereo imagesof FIG. 116 showing the disparity between the images.

FIG. 118 is an illustration of the scene of FIG. 110 showing theoverlaid 3D labels.

FIG. 119 is a flowchart for a depth cue method embodiment of the presentinvention for providing 3D labels.

FIG. 120 is a flowchart for another depth cue method embodiment of thepresent invention for providing 3D labels.

FIG. 121 is a flowchart for yet another depth cue method embodiment ofthe present invention for providing 3D labels.

FIG. 122 is a flowchart for a still another depth cue method embodimentof the present invention for providing 3D labels.

DETAILED DESCRIPTION

The present disclosure relates to eyepiece electro-optics. The eyepiecemay include projection optics suitable to project an image onto asee-through or translucent lens, enabling the wearer of the eyepiece toview the surrounding environment as well as the displayed image. Theprojection optics, also known as a projector, may include an RGB LEDmodule that uses field sequential color. With field sequential color, asingle full color image may be broken down into color fields based onthe primary colors of red, green, and blue and imaged by an LCoS (liquidcrystal on silicon) optical display 210 individually. As each colorfield is imaged by the optical display 210, the corresponding LED coloris turned on. When these color fields are displayed in rapid sequence, afull color image may be seen. With field sequential color illumination,the resulting projected image in the eyepiece can be adjusted for anychromatic aberrations by shifting the red image relative to the blueand/or green image and so on. The image may thereafter be reflected intoa two surface freeform waveguide where the image light engages in totalinternal reflections (TIR) until reaching the active viewing area of thelens where the user sees the image. A processor, which may include amemory and an operating system, may control the LED light source and theoptical display. The projector may also include or be optically coupledto a display coupling lens, a condenser lens, a polarizing beamsplitter, and a field lens.

Referring to FIG. 1, an illustrative embodiment of the augmented realityeyepiece 100 may be depicted. It will be understood that embodiments ofthe eyepiece 100 may not include all of the elements depicted in FIG. 1while other embodiments may include additional or different elements. Inembodiments, the optical elements may be embedded in the arm portions122 of the frame 102 of the eyepiece. Images may be projected with aprojector 108 onto at least one lens 104 disposed in an opening of theframe 102. One or more projectors 108, such as a nanoprojector,picoprojector, microprojector, femtoprojector, LASER-based projector,holographic projector, and the like may be disposed in an arm portion ofthe eyepiece frame 102. In embodiments, both lenses 104 are see-throughor translucent while in other embodiments only one lens 104 istranslucent while the other is opaque or missing. In embodiments, morethan one projector 108 may be included in the eyepiece 100.

In embodiments such as the one depicted in FIG. 1, the eyepiece 100 mayalso include at least one articulating ear bud 120, a radio transceiver118 and a heat sink 114 to absorb heat from the LED light engine, tokeep it cool and to allow it to operate at full brightness. There arealso one or more TI OMAP4 (open multimedia applications processors) 112,and a flex cable with RF antenna 110, all of which will be furtherdescribed herein.

In an embodiment and referring to FIG. 2, the projector 200 may be anRGB projector. The projector 200 may include a housing 202, a heatsink204 and an RGB LED engine or module 206. The RGB LED engine 206 mayinclude LEDs, dichroics, concentrators, and the like. A digital signalprocessor (DSP) (not shown) may convert the images or video stream intocontrol signals, such as voltage drops/current modifications, pulsewidth modulation (PWM) signals, and the like to control the intensity,duration, and mixing of the LED light. For example, the DSP may controlthe duty cycle of each PWM signal to control the average current flowingthrough each LED generating a plurality of colors. A still imageco-processor of the eyepiece may employ noise-filtering, image/videostabilization, and face detection, and be able to make imageenhancements. An audio back-end processor of the eyepiece may employbuffering, SRC, equalization and the like.

The projector 200 may include an optical display 210, such as an LCoSdisplay, and a number of components as shown. In embodiments, theprojector 200 may be designed with a single panel LCoS display 210;however, a three panel display may be possible as well. In the singlepanel embodiment, the display 210 is illuminated with red, blue, andgreen sequentially (aka field sequential color). In other embodiments,the projector 200 may make use of alternative optical displaytechnologies, such as a back-lit liquid crystal display (LCD), afront-lit LCD, a transflective LCD, an organic light emitting diode(OLED), a field emission display (FED), a ferroelectric LCoS (FLCOS),liquid crystal technologies mounted on Sapphire, transparentliquid-crystal micro-displays, quantum-dot displays, and the like.

The eyepiece may be powered by any power supply, such as battery power,solar power, line power, and the like. The power may be integrated inthe frame 102 or disposed external to the eyepiece 100 and in electricalcommunication with the powered elements of the eyepiece 100. Forexample, a solar energy collector may be placed on the frame 102, on abelt clip, and the like. Battery charging may occur using a wallcharger, car charger, on a belt clip, in an eyepiece case, and the like.

The projector 200 may include the LED light engine 206, which may bemounted on heat sink 204 and holder 208, for ensuring vibration-freemounting for the LED light engine, hollow tapered light tunnel 220,diffuser 212 and condenser lens 214. Hollow tunnel 220 helps tohomogenize the rapidly-varying light from the RGB LED light engine. Inone embodiment, hollow light tunnel 220 includes a silvered coating. Thediffuser lens 212 further homogenizes and mixes the light before thelight is led to the condenser lens 214. The light leaves the condenserlens 214 and then enters the polarizing beam splitter (PBS) 218. In thePBS, the LED light is propagated and split into polarization componentsbefore it is refracted to a field lens 216 and the LCoS display 210. TheLCoS display provides the image for the microprojector. The image isthen reflected from the LCoS display and back through the polarizingbeam splitter, and then reflected ninety degrees. Thus, the image leavesmicroprojector 200 in about the middle of the microprojector. The lightthen is led to the coupling lens 504, described below.

FIG. 2 depicts an embodiment of the projector assembly along with othersupporting figures as described herein, but one skilled in the art willappreciate that other configurations and optical technologies may beemployed. For instance, transparent structures, such as with substratesof Sapphire, may be utilized to implement the optical path of theprojector system rather than with reflective optics, thus potentiallyaltering and/or eliminating optical components, such as the beamsplitter, redirecting mirror, and the like. The system may have abacklit system, where the LED RGB triplet may be the light sourcedirected to pass light through the display. As a result the back lightand the display may be mounted either adjacent to the wave guide, orthere may be collumnizing/directing optics after the display to get thelight to properly enter the optic. If there are no directing optics, thedisplay may be mounted on the top, the side, and the like, of thewaveguide. In an example, a small transparent display may be implementedwith a silicon active backplane on a transparent substrate (e.g.sapphire), transparent electrodes controlled by the silicon activebackplane, a liquid crystal material, a polarizer, and the like. Thefunction of the polarizer may be to correct for depolarization of lightpassing through the system to improve the contrast of the display. Inanother example, the system may utilize a spatial light modulator thatimposes some form of spatially-varying modulation on the light path,such as a micro-channel spatial light modulator where a membrane-mirrorlight shutters based on micro-electromechanical systems (MEMS). Thesystem may also utilize other optical components, such as a tunableoptical filter (e.g. with a deformable membrane actuator), a highangular deflection micro-mirror system, a discrete phase opticalelement, and the like.

In other embodiments the eyepiece may utilize OLED displays, quantum-dotdisplays, and the like, that provide higher power efficiency, brighterdisplays, less costly components, and the like. In addition, displaytechnologies such as OLED and quantum-dot displays may allow forflexible displays, and so allowing greater packaging efficiency that mayreduce the overall size of the eyepiece. For example, OLED andquantum-dot display materials may be printed through stamping techniquesonto plastic substrates, thus creating a flexible display component. Forexample, the OLED (organic LED) display may be a flexible, low-powerdisplay that does not require backlighting. It can be curved, as instandard eyeglass lenses. In one embodiment, the OLED display may be orprovide for a transparent display.

Referring to FIG. 82, the eyepiece may utilize a planar illuminationfacility 8208 in association with a reflective display 8210, where lightsource(s) 8202 are coupled 8204 with an edge of the planar illuminationfacility 8208, and where the planar side of the planar illuminationfacility 8208 illuminates the reflective display 8210 that providesimaging of content to be presented to the eye 8222 of the wearer throughtransfer optics 8212. In embodiments, the reflective display 8210 may bean LCD, an LCD on silicon (LCoS), cholesteric liquid crystal, guest-hostliquid crystal, polymer dispersed liquid crystal, phase retardationliquid crystal, and the like, or other liquid crystal technology know inthe art. In other embodiments, the reflective display 8210 may be abi-stable display, such as electrophoretic, electrofluidic,electrowetting, electrokinetic, cholesteric liquid crystal, and thelike, or any other bi-stable display known to the art. The reflectivedisplay 8210 may also be a combination of an LCD technology and abi-stable display technology. In embodiments, the coupling 8204 betweena light source 8202 and the ‘edge’ of the planar illumination facility8208 may be made through other surfaces of the planar illuminationfacility 8208 and then directed into the plane of the planarillumination facility 8208, such as initially through the top surface,bottom surface, an angled surface, and the like. For example, light mayenter the planar illumination facility from the top surface, but into a45° facet such that the light is bent into the direction of the plane.In an alternate embodiment, this bending of direction of the light maybe implemented with optical coatings.

In an example, the light source 8202 may be an RGB LED source (e.g. anLED array) coupled 8204 directly to the edge of the planar illuminationfacility. The light entering the edge of the planar illuminationfacility may then be directed to the reflective display for imaging,such as described herein. Light may enter the reflective display to beimaged, and then redirected back through the planar illuminationfacility, such as with a reflecting surface at the backside of thereflective display. Light may then enter the transfer optics 8212 fordirecting the image to the eye 8222 of the wearer, such as through alens 8214, reflected by a beam splitter 8218 to a reflective surface8220, back through the beam splitter 8218, and the like, to the eye8222. Although the transfer optics 8212 have been described in terms ofthe 8214, 8218, and 8220, it will be appreciated by one skilled in theart that the transfer optics 8212 may include any transfer opticsconfiguration known, including more complex or simpler configurationsthan describe herein. For instance, with a different focal length in thefield lens 8214, the beam splitter 8218 could bend the image directlytowards the eye, thus eliminating the curved mirror 8220, and achievinga simpler design implementation. In embodiments, the light source 8202may be an LED light source, a laser light source, a white light source,and the like, or any other light source known in the art. The lightcoupling mechanism 8204 may be direct coupling between the light source8202 and the planar illumination facility 8208, or through couplingmedium or mechanism, such as a waveguide, fiber optic, light pipe, lens,and the like. The planar illumination facility 8208 may receive andredirect the light to a planar side of its structure through aninterference grating, optical imperfections, scattering features,reflective surfaces, refractive elements, and the like. The planarillumination facility 8208 may be a cover glass over the reflectivedisplay 8210, such as to reduce the combined thickness of the reflectivedisplay 8210 and the planar illumination facility 8208. The planarillumination facility 8208 may further include a diffuser located on theside nearest the transfer optics 8212, to expand the cone angle of theimage light as it passes through the planar illumination facility 8208to the transfer optics 8212. The transfer optics 8212 may include aplurality of optical elements, such as lenses, mirrors, beam splitters,and the like, or any other optical transfer element known to the art.

FIG. 83 presents an embodiment of an optical system 8302 for theeyepiece 8300, where a planar illumination facility 8310 and reflectivedisplay 8308 mounted on substrate 8304 are shown interfacing throughtransfer optics 8212 including an initial diverging lens 8312, a beamsplitter 8314, and a spherical mirror 8318, which present the image tothe eyebox 8320 where the wearer's eye receives the image. In anexample, the flat beam splitter 8314 may be a wire-grid polarizer, ametal partially transmitting mirror coating, and the like, and thespherical reflector 8318 may be a series of dielectric coatings to givea partial mirror on the surface. In another embodiment, the coating onthe spherical mirror 8318 may be a thin metal coating to provide apartially transmitting mirror.

In an embodiment of an optics system, FIG. 84 shows a planarillumination facility 8408 as part of a ferroelectric light-wave circuit(FLC) 8404, including a configuration that utilizes laser light sources8402 coupling to the planar illumination facility 8408 through awaveguide wavelength converter 8420 8422, where the planar illuminationfacility 8408 utilizes a grating technology to present the incominglight from the edge of the planar illumination facility to the planarsurface facing the reflective display 8410. The image light from thereflective display 8410 is then redirected back though the planarillumination facility 8408 though a hole 8412 in the supportingstructure 8414 to the transfer optics. Because this embodiment utilizeslaser light, the FLC also utilizes optical feedback to reduce specklefrom the lasers, by broadening the laser spectrum as described in U.S.Pat. No. 7,265,896. In this embodiment, the laser source 8402 is an IRlaser source, where the FLC combines the beams to RGB, with backreflection that causes the laser light to hop and produce a broadenedbandwidth to provide the speckle suppression. In this embodiment, thespeckle suppression occurs in the wave-guides 8420. The laser light fromlaser sources 8402 is coupled to the planar illumination facility 8408through a multi-mode interference combiner (MMI) 8422. Each laser sourceport is positioned such that the light traversing the MMI combinersuperimposes on one output port to the planar illumination facility8408. The grating of the planar illumination facility 8408 producesuniform illumination for the reflective display. In embodiments, thegrating elements may use a very fine pitch (e.g. interferometric) toproduce the illumination to the reflective display, which is reflectedback with very low scatter off the grating as the light passes throughthe planar illumination facility to the transfer optics. That is, lightcomes out aligned such that the grating is nearly fully transparent.Note that the optical feedback utilized in this embodiment is due to theuse of laser light sources, and when LEDs are utilized, specklesuppression may not be required because the LEDs are already broadbandenough.

In an embodiment of an optics system utilizing a planar illuminationfacility 8502 that includes a configuration with optical imperfections,in this case a ‘grooved’ configuration, is shown in FIG. 85. In thisembodiment, the light source(s) 8202 are coupled 8204 directly to theedge of the planar illumination facility 8502. Light then travelsthrough the planar illumination facility 8502 and encounters smallgrooves 8504A-D in the planar illumination facility material, such asgrooves in a piece of Poly-methyl methacrylate (PMMA). In embodiments,the grooves 8504A-D may vary in spacing as they progress away from theinput port (e.g. less ‘aggressive’ as they progress from 8504A to8504D), vary in heights, vary in pitch, and the like. The light is thenredirected by the grooves 8504A-D to the reflective display 8210 as anincoherent array of light sources, producing fans of rays traveling tothe reflective display 8210, where the reflective display 8210 is farenough away from the grooves 8504A-D to produce illumination patternsfrom each groove that overlap to provide uniform illumination of thearea of the reflective display 8210. In other embodiments, there may bean optimum spacing for the grooves, where the number of grooves perpixel on the reflective display 8210 may be increased to make the lightmore incoherent (more fill), but where in turn this produces lowercontrast in the image provided to the wearer with more grooves tointerfere within the provided image. While this embodiment has beendiscussed with respect to grooves, other optical imperfections, such asdots, are also possible.

In embodiments, and referring to FIG. 86, counter ridges 8604 (or‘anti-grooves’) may be applied into the grooves of the planarillumination facility, such as in a ‘snap-on’ ridge assembly 8602.Wherein the counter ridges 8604 are positioned in the grooves 8504A-Dsuch that there is an air gap between the groove sidewalls and thecounter ridge sidewalls. This air gap provides a defined change inrefractive index as perceived by the light as it travels through theplanar illumination facility that promotes a reflection of the light atthe groove sidewall. The application of counter ridges 8604 reducesaberrations and deflections of the image light caused by the grooves.That is, image light reflected from reflective display 8210 is refractedby the groove sidewall and as such it changes direction because ofSnell's law. By providing counter ridges in the grooves, where thesidewall angle of the groove matches the sidewall angle of the counterridge, the refraction of the image light is compensated for and theimage light is redirected toward the transfer optics 8214.

In embodiments, and referring to FIG. 87, the planar illuminationfacility 8702 may be a laminate structure created out of a plurality oflaminating layers 8704 wherein the laminating layers 8704 havealternating different refractive indices. For instance, the planarillumination facility 8702 may be cut across two diagonal planes 8708 ofthe laminated sheet. In this way, the grooved structure shown in FIGS.85 and 86 is replaced with the laminate structure 8702. For example, thelaminating sheet may be made of similar materials (PMMA 1 versus PMMA2—where the difference is in the molecular weight of the PMMA). As longas the layers are fairly thick, there may be no interference effects,and act as a clear sheet of plastic. In the configuration shown, thediagonal laminations will redirect a small percentage of light source8202 to the reflective display, where the pitch of the lamination isselected to minimize aberration.

In an embodiment of an optics system, FIG. 88 shows a planarillumination facility 8802 utilizing a ‘wedge’ configuration. In thisembodiment, the light source(s) are coupled 8204 directly to the edge ofthe planar illumination facility 8802. Light then travels through theplanar illumination facility 8802 and encounters the slanted surface ofthe first wedge 8804, where the light is redirected to the reflectivedisplay 8210, and then back to the illumination facility 8802 andthrough both the first wedge 8804 and the second wedge 8812 and on tothe transfer optics. In addition, multi-layer coatings 8808 8810 may beapplied to the wedges to improve transfer properties. In an example, thewedge may be made from PMMA, with dimensions of ½ mm high-10 mm width,and spanning the entire reflective display, have 1 to 1.5 degrees angle,and the like. In embodiments, the light may go through multiplereflections within the wedge 8804 before passing through the wedge 8804to illuminate the reflective display 8210. If the wedge 8804 is coatedwith a highly reflecting coating 8808 and 8810, the ray may make manyreflections inside wedge 8804 before turning around and coming back outto the light source 8202 again. However, by employing multi-layercoatings 8808 and 8810 on the wedge 8804, such as with SiO2, NiobiumPentoxide, and the like, light may be directed to illuminate thereflective display 8210. The coatings 8808 and 8810 may be designed toreflect light at a specified wavelength over a wide range of angles, buttransmit light within a certain range of angles (e.g. theta out angles).In embodiments, the design may allow the light to reflect within thewedge until it reaches a transmission window for presentation to thereflective display 8210, where the coating is then configured to enabletransmission. The angle of the wedge directs light from an LED lightingsystem to uniformly irradiate a reflective image display to produce animage that is reflected through the illumination system. By providinglight from the light source 8202 such that a wide cone angle of lightenters the wedge 8804, different rays of light will reach transmissionwindows at different locations along the length of the wedge 8804 sothat uniform illumination of the surface of the reflective display 8210is provided and as a result, the image provided to the wearer's eye hasuniform brightness as determined by the image content in the image.

In embodiments, the see-through optics system including a planarillumination facility 8208 and reflective display 8210 as describedherein may be applied to any head-worn device known to the art, such asincluding the eyepiece as described herein, but also to helmets (e.g.military helmets, pilot helmets, bike helmets, motorcycle helmets, deepsea helmets, space helmets, and the like) ski goggles, eyewear, waterdiving masks, dusk masks, respirators, Hazmat head gear, virtual realityheadgear, simulation devices, and the like. In addition, the opticssystem and protective covering associated with the head-worn device mayincorporate the optics system in a plurality of ways, includinginserting the optics system into the head-worn device in addition tooptics and covering traditionally associated with the head-worn device.For instance, the optics system may be included in a ski goggle as aseparate unit, providing the user with projected content, but where theoptics system doesn't replace any component of the ski goggle, such asthe see-through covering of the ski goggle (e.g. the clear or coloredplastic covering that is exposed to the outside environment, keeping thewind and snow from the user's eyes). Alternatively, the optics systemmay replace, at least in part, certain optics traditionally associatedwith the head-worn gear. For instance, certain optical elements of thetransfer optics 8212 may replace the outer lens of an eyewearapplication. In an example, a beam splitter, lens, or mirror of thetransfer optics 8212 could replace the front lens for an eyewearapplication (e.g. sunglasses), thus eliminating the need for the frontlens of the glasses, such as if the curved reflection mirror 8220 isextended to cover the glasses, eliminating the need for the cover lens.In embodiments, the see-through optics system including a planarillumination facility 8208 and reflective display 8210 may be located inthe head-worn gear so as to be unobtrusive to the function and aestheticof the head-worn gear. For example, in the case of eyewear, or morespecifically the eyepiece, the optics system may be located in proximitywith an upper portion of the lens, such as in the upper portion of theframe.

A planar illumination facility, also know as an illumination module, mayprovide light in a plurality of colors including Red-Green-Blue (RGB)light and/or white light. The light from the illumination module may bedirected to a 3LCD system, a Digital Light Processing (DLP®) system, aLiquid Crystal on Silicon (LCoS) system, or other micro-display ormicro-projection systems. The illumination module may use wavelengthcombining and nonlinear frequency conversion with nonlinear feedback tothe source to provide a source of high-brightness, long-life,speckle-reduced or speckle-free light. Various embodiments of theinvention may provide light in a plurality of colors includingRed-Green-Blue (RGB) light and/or white light. The light from theillumination module may be directed to a 3LCD system, a Digital LightProcessing (DLP) system, a Liquid Crystal on Silicon (LCoS) system, orother micro-display or micro-projection systems. The illuminationmodules described herein may be used in the optical assembly for theeyepiece 100.

One embodiment of the invention includes a system comprising a laser,LED or other light source configured to produce an optical beam at afirst wavelength, a planar lightwave circuit coupled to the laser andconfigured to guide the optical beam, and a waveguide optical frequencyconverter coupled to the planar lightwave circuit, and configured toreceive the optical beam at the first wavelength, convert the opticalbeam at the first wavelength into an output optical beam at a secondwavelength. The system may provide optically coupled feedback which isnonlinearly dependent on the power of the optical beam at the firstwavelength to the laser.

Another embodiment of the invention includes a system comprising asubstrate, a light source, such as a laser diode array or one or moreLEDs disposed on the substrate and configured to emit a plurality ofoptical beams at a first wavelength, a planar lightwave circuit disposedon the substrate and coupled to the light source, and configured tocombine the plurality of optical beams and produce a combined opticalbeam at the first wavelength, and a nonlinear optical element disposedon the substrate and coupled to the planar lightwave circuit, andconfigured to convert the combined optical beam at the first wavelengthinto an optical beam at a second wavelength using nonlinear frequencyconversion. The system may provide optically coupled feedback which isnonlinearly dependent on a power of the combined optical beam at thefirst wavelength to the laser diode array.

Another embodiment of the invention includes a system comprising a lightsource, such as a semiconductor laser array or one or more LEDsconfigured to produce a plurality of optical beams at a firstwavelength, an arrayed waveguide grating coupled to the light source andconfigured to combine the plurality of optical beams and output acombined optical beam at the first wavelength, a quasi-phase matchingwavelength-converting waveguide coupled to the arrayed waveguide gratingand configured to use second harmonic generation to produce an outputoptical beam at a second wavelength based on the combined optical beamat the first wavelength.

Power may be obtained from within a wavelength conversion device and fedback to the source. The feedback power has a nonlinear dependence on theinput power provided by the source to the wavelength conversion device.Nonlinear feedback may reduce the sensitivity of the output power fromthe wavelength conversion device to variations in the nonlinearcoefficients of the device because the feedback power increases if anonlinear coefficient decreases. The increased feedback tends toincrease the power supplied to the wavelength conversion device, thusmitigating the effect of the reduced nonlinear coefficient.

Referring to FIGS. 109A and 109B, a processor 10902 (e.g. a digitalsignal processor) may provide display sequential frames 10924 for imagedisplay through a display component 10928 (e.g. an LCOS displaycomponent) of the eyepiece 100. In embodiments, the sequential frames10924 may be produced with or without a display driver 10912 as anintermediate component between the processor 10902 and the displaycomponent 10928. For example, and referring to FIG. 109A, the processor10902 may include a frame buffer 10904 and a display interface 10908(e.g. a mobile industry processor interface (MIPI), with a displayserial interface (DSI)). The display interface 10908 may provideper-pixel RGB data 10910 to the display driver 10912 as an intermediatecomponent between the processor 10902 and the display component 10928,where the display driver 10912 accepts the per-pixel RGB data 10910 andgenerates individual full frame display data for red 10918, green 10920,and blue 10922, thus providing the display sequential frames 10924 tothe display component 10928. In addition, the display driver 10912 mayprovide timing signals, such as to synchronize the delivery of the fullframes 10918 10920 10922 as display sequential frames 10924 to thedisplay component 10928. In another example, and referring to FIG. 109B,the display interface 10930 may be configured to eliminate the displaydriver 10912 by providing full frame display data for red 10934, green10938, and blue 10940 directly to the display component 10928 as displaysequential frames 10924. In addition, timing signals 10932 may beprovided directly from the display interface 10930 to the displaycomponents. This configuration may provide significantly lower powerconsumption by removing the need for a display driver. Not only may thisdirect panel information remove the need for a driver, but also maysimplify the overall logic of the configuration, and remove redundantmemory required to reform panel information from pixels, to generatepixel information from frame, and the like.

FIG. 89 is a block diagram of an illumination module, according to anembodiment of the invention. Illumination module 8900 comprises anoptical source, a combiner, and an optical frequency converter,according to an embodiment of the invention. An optical source 8902,8904 emits optical radiation 8910, 8914 toward an input port 8922, 8924of a combiner 8906. Combiner 8906 has a combiner output port 8926, whichemits combined radiation 8918. Combined radiation 8918 is received by anoptical frequency converter 8908, which provides output opticalradiation 8928. Optical frequency converter 8908 may also providefeedback radiation 8920 to combiner output port 8926. Combiner 8906splits feedback radiation 8920 to provide source feedback radiation 8912emitted from input port 8922 and source feedback radiation 8916 emittedfrom input port 8924. Source feedback radiation 8912 is received byoptical source 8902, and source feedback radiation 8916 is received byoptical source 8904. Optical radiation 8910 and source feedbackradiation 8912 between optical source 8902 and combiner 8906 maypropagate in any combination of free space and/or guiding structure(e.g., an optical fiber or any other optical waveguide). Opticalradiation 8914, source feedback radiation 8916, combined radiation 8918and feedback radiation 8920 may also propagate in any combination offree space and/or guiding structure.

Suitable optical sources 8902 and 8904 include one or more LEDs or anysource of optical radiation having an emission wavelength that isinfluenced by optical feedback. Examples of sources include lasers, andmay be semiconductor diode lasers. For example, optical sources 8902 and8904 may be elements of an array of semiconductor lasers. Sources otherthan lasers may also be employed (e.g., an optical frequency convertermay be used as a source). Although two sources are shown on FIG. 89, theinvention may also be practiced with more than two sources. Combiner8906 is shown in general terms as a three port device having ports 8922,8924, and 8926. Although ports 8922 and 8924 are referred to as inputports, and port 8926 is referred to as a combiner output port, theseports may be bidirectional and may both receive and emit opticalradiation as indicated above.

Combiner 8906 may include a wavelength dispersive element and opticalelements to define the ports. Suitable wavelength dispersive elementsinclude arrayed waveguide gratings, reflective diffraction gratings,transmissive diffraction gratings, holographic optical elements,assemblies of wavelength-selective filters, and photonic band-gapstructures. Thus, combiner 8906 may be a wavelength combiner, where eachof the input ports i has a corresponding, non-overlapping input portwavelength range for efficient coupling to the combiner output port.

Various optical processes may occur within optical frequency converter8908, including but not limited to harmonic generation, sum frequencygeneration (SFG), second harmonic generation (SHG), difference frequencygeneration, parametric generation, parametric amplification, parametricoscillation, three-wave mixing, four-wave mixing, stimulated Ramanscattering, stimulated Brillouin scattering, stimulated emission,acousto-optic frequency shifting and/or electro-optic frequencyshifting.

In general, optical frequency converter 8908 accepts optical inputs atan input set of optical wavelengths and provides an optical output at anoutput set of optical wavelengths, where the output set differs from theinput set.

Optical frequency converter 8908 may include nonlinear optical materialssuch as lithium niobate, lithium tantalate, potassium titanyl phosphate,potassium niobate, quartz, silica, silicon oxynitride, gallium arsenide,lithium borate, and/or beta-barium borate. Optical interactions inoptical frequency converter 8908 may occur in various structuresincluding bulk structures, waveguides, quantum well structures, quantumwire structures, quantum dot structures, photonic bandgap structures,and/or multi-component waveguide structures.

In cases where optical frequency converter 8908 provides a parametricnonlinear optical process, this nonlinear optical process is preferablyphase-matched. Such phase-matching may be birefringent phase-matching orquasi-phase-matching. Quasi-phase matching may include methods disclosedin U.S. Pat. No. 7,116,468 to Miller, the disclosure of which is herebyincorporated by reference.

Optical frequency converter 8908 may also include various elements toimprove its operation, such as a wavelength selective reflector forwavelength selective output coupling, a wavelength selective reflectorfor wavelength selective resonance, and/or a wavelength selective losselement for controlling the spectral response of the converter.

In embodiments, multiple illumination modules as described in FIG. 89may be associated to form a compound illumination module.

FIG. 90 is a block diagram of an optical frequency converter, accordingto an embodiment of the invention. FIG. 90 illustrates how feedbackradiation 8920 is provided by an exemplary optical frequency converter8908 which provides parametric frequency conversion. Combined radiation8918 provides forward radiation 9002 within optical frequency converter8908 that propagates to the right on FIG. 90, and parametric radiation9004, also propagating to the right on FIG. 90, is generated withinoptical frequency converter 8908 and emitted from optical frequencyconverter 8908 as output optical radiation 8928. Typically there is anet power transfer from forward radiation 9002 to parametric radiation9004 as the interaction proceeds (i.e., as the radiation propagates tothe right in this example). A reflector 9008, which may havewavelength-dependent transmittance, is disposed in optical frequencyconverter 8908 to reflect (or partially reflect) forward radiation 9002to provide backward radiation 9006 or may be disposed externally tooptical frequency converter 8908 after endface 9010. Reflector 9008 maybe a grating, an internal interface, a coated or uncoated endface, orany combination thereof. The preferred level of reflectivity forreflector 9008 is greater than 90%. A reflector located at an inputinterface 9012 provides purely linear feedback (i.e., feedback that doesnot depend on the process efficiency). A reflector located at an endface9010 provides a maximum degree of nonlinear feedback, since thedependence of forward power on process efficiency is maximized at theoutput interface (assuming a phase-matched parametric interaction).

FIG. 91 is a block diagram of a laser illumination module, according toan embodiment of the invention. While lasers are used in thisembodiment, it is understood that other light sources, such as LEDs, mayalso be used. Laser illumination module 9100 comprises an array of diodelasers 9102, waveguides 9104 and 9106, star couplers 9108 and 9110 andoptical frequency converter 9114. An array of diode lasers 9102 haslasing elements coupled to waveguides 9104 acting as input ports (suchas ports 8922 and 8924 on FIG. 89) to a planar waveguide star coupler9108. Star coupler 9108 is coupled to another planar waveguide starcoupler 9110 by waveguides 9106 which have different lengths. Thecombination of star couplers 9108 and 9110 with waveguides 9106 may bean arrayed waveguide grating, and acts as a wavelength combiner (e.g.,combiner 8906 on FIG. 89) providing combined radiation 8918 to waveguide9112. Waveguide 9112 provides combined radiation 8918 to opticalfrequency converter 9114. Within optical frequency converter 9114, anoptional reflector 9116 provides a back reflection of combined radiation8918. As indicated above in connection with FIG. 90, this backreflection provides nonlinear feedback according to embodiments of theinvention. One or more of the elements described with reference to FIG.91 may be fabricated on a common substrate using planar coating methodsand/or lithography methods to reduce cost, parts count and alignmentrequirements.

A second waveguide may be disposed such that its core is in closeproximity with the core of the waveguide in optical frequency converter8908. As is known in the art, this arrangement of waveguides functionsas a directional coupler, such that radiation in waveguide may provideadditional radiation in optical frequency converter 8908. Significantcoupling may be avoided by providing radiation at wavelengths other thanthe wavelengths of forward radiation 9002 or additional radiation may becoupled into optical frequency converter 8908 at a location whereforward radiation 9002 is depleted.

While standing wave feedback configurations where the feedback powerpropagates backward along the same path followed by the input power areuseful, traveling wave feedback configurations may also be used. In atraveling wave feedback configuration, the feedback re-enters the gainmedium at a location different from the location at which the inputpower is emitted from.

FIG. 92 is a block diagram of a compound laser illumination module,according to another embodiment of the invention. Compound laserillumination module 9200 comprises one or more laser illuminationmodules 9100 described with reference to FIG. 91. Although FIG. 92illustrates compound laser illumination module 9200 including threelaser illumination modules 9100 for simplicity, compound laserillumination module 9200 may include more or fewer laser illuminationmodules 9100. An array of diode lasers 9210 may include one or morearrays of diode lasers 9102 which may be an array of laser diodes, adiode laser array, and/or a semiconductor laser array configured to emitoptical radiation within the infrared spectrum, i.e., with a wavelengthshorter than radio waves and longer than visible light.

Laser array output waveguides 9220 couple to the diode lasers in thearray of diode lasers 9210 and directs the outputs of the array of diodelasers 9210 to star couplers 9108A-C. The laser array output waveguides9220, the arrayed waveguide gratings 9230, and the optical frequencyconverters 9114A-C may be fabricated on a single substrate using aplanar lightwave circuit, and may comprise silicon oxynitride waveguidesand/or lithium tantalate waveguides.

Arrayed waveguide gratings 9230 comprise the star couplers 9108A-C,waveguides 9106A-C, and star couplers 9110A-C. Waveguides 9112A-Cprovide combined radiation to optical frequency converters 9114A-C andfeedback radiation to star couplers 9110A-C, respectively.

Optical frequency converters 9114A-C may comprise nonlinear optical(NLO) elements, for example optical parametric oscillator elementsand/or quasi-phase matched optical elements.

Compound laser illumination module 9200 may produce output opticalradiation at a plurality of wavelengths. The plurality of wavelengthsmay be within a visible spectrum, i.e., with a wavelength shorter thaninfrared and longer than ultraviolet light. For example, waveguide 9240Amay similarly provide output optical radiation between about 450 nm andabout 470 nm, waveguide 9240B may provide output optical radiationbetween about 525 nm and about 545 nm, and waveguide 9240C may provideoutput optical radiation between about 615 nm and about 660 nm. Theseranges of output optical radiation may again be selected to providevisible wavelengths (for example, blue, green and red wavelengths,respectively) that are pleasing to a human viewer, and may again becombined to produce a white light output.

The waveguides 9240A-C may be fabricated on the same planar lightwavecircuit as the laser array output waveguides 9220, the arrayed waveguidegratings 9230, and the optical frequency converters 9114A-C. In someembodiments, the output optical radiation provided by each of thewaveguides 9240A-C may provide an optical power in a range betweenapproximately 1 watts and approximately 20 watts.

The optical frequency converter 9114 may comprise a quasi-phase matchingwavelength-converting waveguide configured to perform second harmonicgeneration (SHG) on the combined radiation at a first wavelength, andgenerate radiation at a second wavelength. A quasi-phase matchingwavelength-converting waveguide may be configured to use the radiationat the second wavelength to pump an optical parametric oscillatorintegrated into the quasi-phase matching wavelength-converting waveguideto produce radiation at a third wavelength, the third wavelengthoptionally different from the second wavelength. The quasi-phasematching wavelength-converting waveguide may also produce feedbackradiation propagated via waveguide 9112 through the arrayed waveguidegrating 9230 to the array of diode lasers 9210, thereby enabling eachlaser disposed within the array of diode lasers 9210 to operate at adistinct wavelength determined by a corresponding port on the arrayedwaveguide grating.

For example, compound laser illumination module 9200 may be configuredusing an array of diode lasers 9210 nominally operating at a wavelengthof approximately 830 nm to generate output optical radiation in avisible spectrum corresponding to any of the colors red, green, or blue.

Compound laser illumination module 9200 may be optionally configured todirectly illuminate spatial light modulators without intervening optics.In some embodiments, compound laser illumination module 9200 may beconfigured using an array of diode lasers 9210 nominally operating at asingle first wavelength to simultaneously produce output opticalradiation at multiple second wavelengths, such as wavelengthscorresponding to the colors red, green, and blue. Each different secondwavelength may be produced by an instance of laser illumination module9100.

The compound laser illumination module 9200 may be configured to producediffraction-limited white light by combining output optical radiation atmultiple second wavelengths into a single waveguide using, for example,waveguide-selective taps (not shown).

The array of diode lasers 9210, laser array output waveguides 9220,arrayed waveguide gratings 9230, waveguides 9112, optical frequencyconverters 9114, and frequency converter output waveguides 9240 may befabricated on a common substrate using fabrication processes such ascoating and lithography. The beam shaping element 9250 is coupled to thecompound laser illumination module 9200 by waveguides 9240A-C, describedwith reference to FIG. 92.

Beam shaping element 9250 may be disposed on a same substrate as thecompound laser illumination module 9200. The substrate may, for example,comprise a thermally conductive material, a semiconductor material, or aceramic material. The substrate may comprise copper-tungsten, silicon,gallium arsenide, lithium tantalate, silicon oxynitride, and/or galliumnitride, and may be processed using semiconductor manufacturingprocesses including coating, lithography, etching, deposition, andimplantation.

Some of the described elements, such as the array of diode lasers 9210,laser array output waveguides 9220, arrayed waveguide gratings 9230,waveguides 9112, optical frequency converters 9114, waveguides 9240,beam shaping element 9250, and various related planar lightwave circuitsmay be passively coupled and/or aligned, and in some embodiments,passively aligned by height on a common substrate. Each of thewaveguides 9240A-C may couple to a different instance of beam shapingelement 9250, rather than to a single element as shown.

Beam shaping element 9250 may be configured to shape the output opticalradiation from waveguides 9240A-C into an approximately rectangulardiffraction-limited optical beam, and may further configure the outputoptical radiation from waveguides 9240A-C to have a brightnessuniformity greater than approximately 95% across the approximatelyrectangular beam shape.

The beam shaping element 9250 may comprise an aspheric lens, such as a“top-hat” microlens, a holographic element, or an optical grating. Insome embodiments, the diffraction-limited optical beam output by thebeam shaping element 9250 produces substantially reduced or no speckle.The optical beam output by the beam shaping element 9250 may provide anoptical power in a range between approximately 1 watt and approximately20 watts, and a substantially flat phase front.

FIG. 93 is a block diagram of an imaging system, according to anembodiment of the invention. Imaging system 9300 comprises light engine9310, optical beams 9320, spatial light modulator 9330, modulatedoptical beams 9340, and projection lens 9350. The light engine 9310 maybe a compound optical illumination module, such as multiple illuminationmodules described in FIG. 89, a compound laser illumination module 9200,described with reference to FIG. 92, or a laser illumination system9300, described with reference to FIG. 93. Spatial light modulator 9330may be a 3LCD system, a DLP system, a LCoS system, a transmissive liquidcrystal display (e.g. transmissive LCoS), a liquid-crystal-on-siliconarray, a grating-based light valve, or other micro-display ormicro-projection system or reflective display.

The spatial light modulator 9330 may be configured to spatially modulatethe optical beam 9320. The spatial light modulator 9330 may be coupledto electronic circuitry configured to cause the spatial light modulator9330 to modulate a video image, such as may be displayed by a televisionor a computer monitor, onto the optical beam 9320 to produce a modulatedoptical beam 9340. In some embodiments, modulated optical beam 9340 maybe output from the spatial light modulator on a same side as the spatiallight modulator receives the optical beam 9320, using optical principlesof reflection. In other embodiments, modulated optical beam 9340 may beoutput from the spatial light modulator on an opposite side as thespatial light modulator receives the optical beam 9320, using opticalprinciples of transmission. The modulated optical beam 9340 mayoptionally be coupled into a projection lens 9350. The projection lens9350 is typically configured to project the modulated optical beam 9340onto a display, such as a video display screen.

A method of illuminating a video display may be performed using acompound illumination module such as one comprising multipleillumination modules 8900, a compound laser illumination module 9100, alaser illumination system 9200, or an imaging system 9300. Adiffraction-limited output optical beam is generated using a compoundillumination module, compound laser illumination module 9100, laserillumination system 9200 or light engine 9310. The output optical beamis directed using a spatial light modulator, such as spatial lightmodulator 9330, and optionally projection lens 9350. The spatial lightmodulator may project an image onto a display, such as a video displayscreen.

The illumination module may be configured to emit any number ofwavelengths including one, two, three, four, five, six, or more, thewavelengths spaced apart by varying amounts, and having equal or unequalpower levels. An illumination module may be configured to emit a singlewavelength per optical beam, or multiple wavelengths per optical beam.An illumination module may also comprise additional components andfunctionality including polarization controller, polarization rotator,power supply, power circuitry such as power FETs, electronic controlcircuitry, thermal management system, heat pipe, and safety interlock.In some embodiments, an illumination module may be coupled to an opticalfiber or a lightguide, such as glass (e.g. BK7).

Some options for an LCoS front light design include: 1) Wedge withMultiLayer Coating (MLC). This concept uses MLC to define specificreflected and transmitted angles; 2) Wedge with polarized beamsplittercoating. This concept works like a regular PBS Cube, but at a muchshallower angle. This can be PBS coating or a wire grid film; 3) PBSPrism bars (these are similar to Option #2) but have a seam down thecenter of the panel; and 4) Wire Grid Polarizer plate beamsplitter(similar to the PBS wedge, but just a plate, so that it is mostly airinstead of solid glass).

FIG. 95 depicts an embodiment of an LCoS front light design. In thisembodiment, light from an RGB LED 9508 illuminates a front light 9504,which can be a wedge, PBS, and the like. The light strikes a polarizer9510 and is transmitted in its S state to an LCoS 9502 where it getsreflected as image light in its P state back through an asphere 9512. Aninline polarizer 9514 may polarize the image light again and/or cause a½ wave rotation to the S state. The image light then hits a wire gridpolarizer 9520 and reflects to a curved (spherical) partial mirror 9524,passing through a ½ wave retarder 9522 on its way. The image lightreflects from the mirror to the user's eye 9518, once more traversingthe ½ wave retarder 9522 and wire grid polarizer 9520. Various examplesof the front light 9504 will now be described.

In embodiments, the optical assembly includes a partially reflective,partially transmitting optical element that reflects respective portionsof image light from the image source and transmits scene light from asee-through view of the surrounding environment, so that a combinedimage comprised of portions of the reflected image light and thetransmitted scene light is provided to a user's eye.

FIG. 96 depicts an embodiment of a front light 9504 comprising opticallybonded prisms with a polarizer. The prisms appear as two rectangularsolids with a substantially transparent interface 9602 between the two.Each rectangular is diagonally bisected and a polarizing coating 9604 isdisposed along the interface of the bisection. The lower triangle formedby the bisected portion of the rectangular solid may optionally be madeas a single piece 9608. The prisms may be made from BK-7 or theequivalent. In this embodiment, the rectangular solids have square endsthat measure 2 mm by 2 mm The length of the solids in this embodiment is10 mm. In an alternate embodiment, the bisection comprises a 50% mirror9704 surface and the interface between the two rectangular solidscomprises a polarizer 9702 that may pass light in the P state.

FIG. 98 depicts three versions of an LCoS front light design. FIG. 98Adepicts a wedge with MultiLayer Coating (MLC). This concept uses MLC todefine specific reflected and transmitted angles. In this embodiment,image light of either P or S polarization state is observed by theuser's eye. FIG. 98B depicts a PBS with a polarizer coating. Here, onlyS-polarized image light is transmitted to the user's eye. FIG. 98Cdepicts a right angle prism, eliminating much of the material of theprism enabling the image light to be transmitted through air asS-polarized light.

FIG. 99 depicts a wedge plus PBS with a polarizing coating 9902 layeredon an LCoS 9904.

FIG. 100 depicts two embodiments of prisms with light entering the shortend (A) and light entering along the long end (B). In FIG. 100A, a wedgeis formed by offset bisecting a rectangular solid to form at least one8.6 degree angle at the bisect interface. In this embodiment, the offsetbisection results in a segment that is 0.5 mm high and another that is1.5 mm on the side through which the RGB LEDs 10002 are transmittinglight. Along the bisection, a polarizing coating 10004 is disposed. InFIG. 100B, a wedge is formed by offset bisecting a rectangular solid toform at least one 14.3 degree angle at the bisect interface. In thisembodiment, the offset bisection results in a segment that is 0.5 mmhigh and another that is 1.5 mm on the side through which the RGB LEDs10008 are transmitting light. Along the bisection, a polarizing coating10010 is disposed.

FIG. 101 depicts a curved PBS film 10104 illuminated by an RGB LED 10102disposed over an LCoS chip 10108. The PBS film 10104 reflects the RGBlight from the LED array 10102 onto the LCOS chip's surface 10108, butlets the light reflected from the imaging chip pass through unobstructedto the optical assembly and eventually to the user's eye. Films used inthis system include Asahi Film, which is a Tri-Acetate Cellulose orcellulose acetate substrate (TAC). In embodiments, the film may have UVembossed corrugations at 100 nm and a calendared coating built up onridges that can be angled for incidence angle of light. The Asahi filmmay come in rolls that are 20 cm wide by 30 m long and has BEFproperties when used in LCD illumination. The Asahi film may supportwavelengths from visible through IR and may be stable up to 100° C.

In another embodiment, FIGS. 21 and 22 depict an alternate arrangementof the waveguide and projector in exploded view. In this arrangement,the projector is placed just behind the hinge of the arm of the eyepieceand it is vertically oriented such that the initial travel of the RGBLED signals is vertical until the direction is changed by a reflectingprism in order to enter the waveguide lens. The vertically arrangedprojection engine may have a PBS 218 at the center, the RGB LED array atthe bottom, a hollow, tapered tunnel with thin film diffuser to mix thecolors for collection in an optic, and a condenser lens. The PBS mayhave a pre-polarizer on an entrance face. The pre-polarizer may bealigned to transmit light of a certain polarization, such as p-polarizedlight and reflect (or absorb) light of the opposite polarization, suchas s-polarized light. The polarized light may then pass through the PBSto the field lens 216. The purpose of the field lens 216 may be tocreate near telecentric illumination of the LCoS panel. The LCoS displaymay be truly reflective, reflecting colors sequentially with correcttiming so the image is displayed properly. Light may reflect from theLCoS panel and, for bright areas of the image, may be rotated tos-polarization. The light then may refract through the field lens 216and may be reflected at the internal interface of the PBS and exit theprojector, heading toward the coupling lens. The hollow, tapered tunnel220 may replace the homogenizing lenslet from other embodiments. Byvertically orienting the projector and placing the PBS in the center,space is saved and the projector is able to be placed in a hinge spacewith little moment arm hanging from the waveguide.

Light reflected or scattered from the image source or associated opticsof the eyepiece may pass outward into the environment. These lightlosses are perceived by external viewers as ‘eyeglow’ or ‘night glow’where portions of the lenses or the areas surrounding the eyepieceappear to be glowing when viewed in a dimly lit environment. In certaincases of eyeglow as shown in FIG. 22A, the displayed image can be seenas an observable image 2202A in the display areas when viewed externallyby external viewers. To maintain privacy of the viewing experience forthe user both in terms of maintaining privacy of the images being viewedand in terms of making the user less noticeable when using the eyepiecein a dimly lit environment, it is preferable to reduce eyeglow. Methodsand apparatus may reduce eyeglow through a light control element, suchas with a partially reflective mirror in the optics associated with theimage source, with polarizing optics, and the like. For instance, lightentering the waveguide may be polarized, such as s-polarized. The lightcontrol element may include a linear polarizer. Wherein the linearpolarizer in the light control element is oriented relative to thelinearly polarized image light so that the second portion of thelinearly polarized image light that passes through the partiallyreflecting mirror is blocked and eyeglow is reduced. In embodiments,eyeglow may be minimized or eliminated by attaching lenses to thewaveguide or frame, such as the snap-fit optics described herein, thatare oppositely polarized from the light reflecting from the user's eye,such as p-polarized in this case.

In embodiments, the light control element may include a second quarterwave film and a linear polarizer. Wherein the second quarter wave filmconverts a second portion of a circularly polarized image light intolinearly polarized image light with a polarization state that is blockedby the linear polarizer in the light control element so that eyeglow isreduced. For example, when the light control element includes a linearpolarizer and a quarter wave film, incoming unpolarized scene light fromthe external environment in front of the user is converted to linearlypolarized light while 50% of the light is blocked. The first portion ofscene light that passes through the linear polarizer is linearlypolarized light which is converted by the quarter wave film tocircularly polarized light. The third portion of scene light that isreflected from the partially reflecting mirror has reversed circularpolarization which is then converted to linearly polarized light by thesecond quarter wave film. The linear polarizer then blocks the reflectedthird portion of the scene light thereby reducing escaping light andreducing eyeglow. FIG. 22B shows an example of a see-through displayassembly with a light control element in a glasses frame. The glassescross-section 2200B shows the components of see-through display assemblyin a glasses frame 2202B. Wherein, the light control element covers theentire see-through view seen by the user. Supporting members 2204B and2208B are shown supporting the partially reflecting mirror 2210B and thebeam splitter layer 2212B respectively in the field of view of theuser's eye 2214B. The supporting members 2204B and 2208B along with thelight control element 2218B are connected to the glasses frame 2202B.The other components such as the folding mirror 2220B and the firstquarter wave film 2222B are also connected to the supporting members2204B and 2208B so that the combined assembly is structurally sound.

In an embodiment, an absorptive polarizer in the optical assembly isused to reduce stray light. The absorptive polarizer may include ananti-reflective coating. The absorptive polarizer may be disposed aftera focusing lens of the optical assembly to reduce light passing throughan optically flat film of the optical assembly. The light from the imagesource may be polarized to increase contrast.

In an embodiment, an anti-reflective coating in the optical assembly maybe used to reduce stray light. The anti-reflective coating may bedisposed on a polarizer of the optical assembly or a retarding film ofthe optical assembly. The retarding film may be a quarter wave film or ahalf wave film. The anti-reflective coating may be disposed on an outersurface of a partially reflecting mirror. The light from the imagesource may be polarized to increase contrast.

Referring to FIG. 102, an image source 10228 directs image light to abeam splitter layer of the optical assembly. FIG. 103 depicts a blow-upof the image source 10228. In this particular embodiment, the imagesource 10228 is shown containing a light source (LED Bar 10302) thatdirects light through a diffuser 10304 and prepolarizer 10308 to acurved wire grid polarizer 10310 where the light is reflected to an LCoSdisplay 10312. Image light from the LCoS is then reflected back throughthe curved wire grid polarizer 10310 and a half wave film 10312 to thebeam splitter layer of the optical assembly 10200.

Referring to FIG. 104, LEDs provide unpolarized light. The diffuserspreads and homogenizes the light from the LEDs. The absorptiveprepolarizer converts the light to S polarization. The S polarized lightis then reflected toward the LCOS by the curved wire grid polarizer. TheLCOS reflects the S polarized light and converts it to P polarized lightdepending on local image content. The P polarized light passes throughthe curved wire grid polarizer becoming P polarized image light. Thehalf wave film converts the P polarized image light to S polarized imagelight.

Referring again to FIG. 102, the beam splitter layer 10204 is apolarizing beam splitter, or the image source provides polarized imagelight 10208 and the beam splitter layer 10204 is a polarizing beamsplitter, so that the reflected image light 10208 is linearly polarizedlight, this embodiment and the associated polarization control is shownin FIG. 102. For the case where the image source provides linearlypolarized image light and the beam splitter layer 10204 is a polarizingbeam splitter, the polarization state of the image light is aligned tothe polarizing beam splitter so that the image light 10208 is reflectedby the polarizing beam splitter. FIG. 102 shows the reflected imagelight as having S state polarization. In cases where the beam splitterlayer 10204 is a polarizing beam splitter, a first quarter wave film10210 is provided between the beam splitter layer 10204 and thepartially reflecting mirror 10212. The first quarter wave film 10210converts the linearly polarized image light to circularly polarizedimage light (shown as S being converted to CR in FIG. 102). Thereflected first portion of image light 10208 is then also circularlypolarized where the circular polarization state is reversed (shown as CLin FIG. 102) so that after passing back through the quarter wave film,the polarization state of the reflected first portion of image light10208 is reversed (to P polarization) compared to the polarization stateof the image light 10208 provided by the image source (shown as S). As aresult, the reflected first portion of the image light 10208 passesthrough the polarizing beam splitter without reflection losses. When thebeam splitter layer 10204 is a polarizing beam splitter and thesee-through display assembly 10200 includes a first quarter wave film10210, the light control element 10230 is a second quarter wave film anda linear polarizer 10220. In embodiments, the light control element10230 includes a controllable darkening layer 10214. Wherein the secondquarter wave film 10218 converts the second portion of the circularlypolarized image light 10208 into linearly polarized image light 10208(shown as CR being converted to S) with a polarization state that isblocked by the linear polarizer 10220 in the light control element 10230so that eyeglow is reduced.

When the light control element 10230 includes a linear polarizer 10220and a quarter wave film 10218, incoming unpolarized scene light 10222from the external environment in front of the user is converted tolinearly polarized light (shown as P polarization state in FIG. 102)while 50% of the light is blocked. The first portion of scene light10222 that passes through the linear polarizer 10220 is linearlypolarized light which is converted by the quarter wave film tocircularly polarized light (shown as P being converted to CL in FIG.102). The third portion of scene light that is reflected from thepartially reflecting mirror 10212 has reversed circular polarization(shown as converting from CL to CR in FIG. 102) which is then convertedto linearly polarized light by the second quarter wave film 10218 (shownas CR converting to S polarization in FIG. 102). The linear polarizer10220 then blocks the reflected third portion of the scene light therebyreducing escaping light and reducing eyeglow.

As shown in FIG. 102, the reflected first portion of image light 10208and the transmitted second portion of scene light have the same circularpolarization state (shown as CL) so that they combine and are convertedby the first quarter wave film 10210 into linearly polarized light(shown as P) which passes through the beam splitter when the beamsplitter layer 10204 is a polarizing beam splitter. The linearlypolarized combined light 10224 then provides a combined image to theuser's eye 10202 located at the back of the see-through display assembly10200, where the combined image is comprised of overlaid portions of thedisplayed image from the image source and the see-through view of theexternal environment in front of the user.

The beamsplitter layer 10204 includes an optically flat film, such asthe Asahi TAC film discussed herein. The beamsplitter layer 10204 may bedisposed at an angle in front of a user's eye so that it reflects andtransmits respective portions of image light and transmits scene lightfrom a see-through view of the surrounding environment, so that acombined image comprised of portions of the image light and thetransmitted scene light is provided to a user's eye. The optically flatfilm may be a wire grid polarizer. The optically flat film may belaminated to a transparent substrate. The optically flat film may bemolded into a surface of the eyepiece. The optically flat film may bepositioned at less then 40 degrees from vertical.

In an embodiment, the components in FIG. 102 collectively form anelectro-optic module. The angle of the optical axis associated with thedisplay may be 10 degrees or more forward of vertical. This degree oftilt refers to how the upper part of the optics module leans forward.This allows the beamsplitter angle to be reduced which makes the opticsmodule thinner.

The ratio of the height of the curved polarizing film to the width ofthe reflective image display is less than 1:1. The curve on thepolarizing film determines the width of the illuminated area on thereflective display, and the tilt of the curved area determines thepositioning of the illuminated area on the reflective display. Thecurved polarizing film reflects illumination light of a firstpolarization state onto the reflective display, which changes thepolarization of the illumination light and generates image light, andthe curved polarizing film passes reflected image light. The curvedpolarizing film includes a portion that is parallel to the reflectivedisplay over the light source. The height of the image source may be atleast 80% of the display active area width, at least 3.5 mm, or lessthan 4 mm

Referring to FIGS. 105 A through C, the angle of the curved wire gridpolarizer controls the direction of the image light. The curve of thecurved wire grid polarizer controls the width of the image light. Thecurve enables use of a narrow light source because it spreads the lightwhen the light strikes it and then folds it/reflects it to uniformlyilluminate an image display. Image light passing back through the wiregrid polarizer is unperturbed. Thus, the curve also enables theminiaturization of the optical assembly.

In FIGS. 21-22, augmented reality eyepiece 2100 includes a frame 2102and left and right earpieces or temple pieces 2104. Protective lenses2106, such as ballistic lenses, are mounted on the front of the frame2102 to protect the eyes of the user or to correct the user's view ofthe surrounding environment if they are prescription lenses. The frontportion of the frame may also be used to mount a camera or image sensor2130 and one or more microphones 2132. Not visible in FIG. 21,waveguides are mounted in the frame 2102 behind the protective lenses2106, one on each side of the center or adjustable nose bridge 2138. Thefront cover 2106 may be interchangeable, so that tints or prescriptionsmay be changed readily for the particular user of the augmented realitydevice. In one embodiment, each lens is quickly interchangeable,allowing for a different prescription for each eye. In one embodiment,the lenses are quickly interchangeable with snap-fits as discussedelsewhere herein. Certain embodiments may only have a projector andwaveguide combination on one side of the eyepiece while the other sidemay be filled with a regular lens, reading lens, prescription lens, orthe like. The left and right ear pieces 2104 may each vertically mount aprojector or microprojector 2114 or other image source atop aspring-loaded hinge 2128 for easier assembly and vibration/shockprotection. Each temple piece also includes a temple housing 2116 formounting associated electronics for the eyepiece, and each may alsoinclude an elastomeric head grip pad 2120, for better retention on theuser. Each temple piece also includes extending, wrap-around ear buds2112 and an orifice 2126 for mounting a headstrap 2142.

As noted, the temple housing 2116 contains electronics associated withthe augmented reality eyepiece. The electronics may include severalcircuit boards, as shown, such as for the microprocessor and radios2122, the communications system on a chip (SOC) 2124, and the openmultimedia applications processor (OMAP) processor board 2140. Thecommunications system on a chip (SOC) may include electronics for one ormore communications capabilities, including a wide local area network(WLAN), BlueTooth™ communications, frequency modulation (FM) radio, aglobal positioning system (GPS), a 3-axis accelerometer, one or moregyroscopes, and the like. In addition, the right temple piece mayinclude an optical trackpad (not shown) on the outside of the templepiece for user control of the eyepiece and one or more applications.

In an embodiment, a digital signal processor (DSP) may be programmedand/or configured to receive video feed information and configure thevideo feed to drive whatever type of image source is being used with theoptical display. The DSP may include a bus or other communicationmechanism for communicating information, and an internal processorcoupled with the bus for processing the information. The DSP may includea memory, such as a random access memory (RAM) or other dynamic storagedevice (e.g., dynamic RAM (DRAM), static RAM (SRAM), and synchronousDRAM (SDRAM)), coupled to the bus for storing information andinstructions to be executed. The DSP can include a non-volatile memorysuch as for example a read only memory (ROM) or other static storagedevice (e.g., programmable ROM (PROM), erasable PROM (EPROM), andelectrically erasable PROM (EEPROM)) coupled to the bus for storingstatic information and instructions for the internal processor. The DSPmay include special purpose logic devices (e.g., application specificintegrated circuits (ASICs)) or configurable logic devices (e.g., simpleprogrammable logic devices (SPLDs), complex programmable logic devices(CPLDs), and field programmable gate arrays (FPGAs)).

The DSP may include at least one computer readable medium or memory forholding instructions programmed and for containing data structures,tables, records, or other data necessary to drive the optical display.Examples of computer readable media suitable for applications of thepresent disclosure may be compact discs, hard disks, floppy disks, tape,magneto-optical disks, PROMs (EPROM, EEPROM, flash EPROM), DRAM, SRAM,SDRAM, or any other magnetic medium, compact discs (e.g., CD-ROM), orany other optical medium, punch cards, paper tape, or other physicalmedium with patterns of holes, a carrier wave (described below), or anyother medium from which a computer can read. Various forms of computerreadable media may be involved in carrying out one or more sequences ofone or more instructions to the optical display for execution. The DSPmay also include a communication interface to provide a datacommunication coupling to a network link that can be connected to, forexample, a local area network (LAN), or to another communicationsnetwork such as the Internet. Wireless links may also be implemented. Inany such implementation, an appropriate communication interface can sendand receive electrical, electromagnetic or optical signals that carrydigital data streams representing various types of information (such asthe video information) to the optical display.

In embodiments, the eyepiece may provide an external interface tocomputer peripheral devices, such as a monitor, display, TV, keyboards,mice, memory storage (e.g. external hard drive, optical drive, solidstate memory), network interface (e.g. to the Internet), and the like.For instance, the external interface may provide direct connectivity toexternal computer peripheral devices (e.g. connect directly to amonitor), indirect connectivity to external computer peripheral devices(e.g. through a central external peripheral interface device), through awired connection, though a wireless connection, and the like. In anexample, the eyepiece may be able to connect to a central externalperipheral interface device that provides connectivity to externalperipheral devices, where the external peripheral interface device mayinclude computer interface facilities, such as a computer processor,memory, operating system, peripheral drivers and interfaces, USB port,external display interface, network port, speaker interface, microphoneinterface, and the like. In embodiments, the eyepiece may be connectedto the central external peripheral interface by a wired connection,wireless connection, directly in a cradle, and the like, and whenconnected may provide the eyepiece with computational facilities similarto or identical to a personal computer.

The frame 2102 is in a general shape of a pair of wrap-aroundsunglasses. The sides of the glasses include shape-memory alloy straps2134, such as nitinol straps. The nitinol or other shape-memory alloystraps are fitted for the user of the augmented reality eyepiece. Thestraps are tailored so that they assume their trained or preferred shapewhen worn by the user and warmed to near body temperature. Inembodiments, the fit of the eyepiece may provide user eye widthalignment techniques and measurements. For instance, the position and/oralignment of the projected display to the wearer of the eyepiece may beadjustable in position to accommodate the various eye widths of thedifferent wearers. The positioning and/or alignment may be automatic,such as though detection of the position of the wearer's eyes throughthe optical system (e.g. iris or pupil detection), or manual, such as bythe wearer, and the like.

Other features of this embodiment include detachable, noise-cancellingearbuds. As seen in the figure, the earbuds are intended for connectionto the controls of the augmented reality eyepiece for delivering soundsto ears of the user. The sounds may include inputs from the wirelessinternet or telecommunications capability of the augmented realityeyepiece. The earbuds also include soft, deformable plastic or foamportions, so that the inner ears of the user are protected in a mannersimilar to earplugs. In one embodiment, the earbuds limit inputs to theuser's ears to about 85 dB. This allows for normal hearing by thewearer, while providing protection from gunshot noise or other explosivenoises and listening in high background noise environments. In oneembodiment, the controls of the noise-cancelling earbuds have anautomatic gain control for very fast adjustment of the cancellingfeature in protecting the wearer's ears.

FIG. 23 depicts a layout of the vertically arranged projector 2114 in aneyepiece 2300, where the illumination light passes from bottom to topthrough one side of the PBS on its way to the display and imager board,which may be silicon backed, and being refracted as image light where ithits the internal interfaces of the triangular prisms which constitutethe polarizing beam splitter, and is reflected out of the projector andinto the waveguide lens. In this example, the dimensions of theprojector are shown with the width of the imager board being 11 mm, thedistance from the end of the imager board to the image centerline being10.6 mm, and the distance from the image centerline to the end of theLED board being about 11.8 mm

A detailed and assembled view of the components of the projectordiscussed above may be seen in FIG. 25. This view depicts how compactthe micro-projector 2500 is when assembled, for example, near a hinge ofthe augmented reality eyepiece. Microprojector 2500 includes a housingand a holder 2508 for mounting certain of the optical pieces. As eachcolor field is imaged by the optical display 2510, the corresponding LEDcolor is turned on. The RGB LED light engine 2502 is depicted near thebottom, mounted on heat sink 2504. The holder 2508 is mounted atop theLED light engine 2502, the holder mounting light tunnel 2520, diffuserlens 2512 (to eliminate hotspots) and condenser lens 2514. Light passesfrom the condenser lens into the polarizing beam splitter 2518 and thento the field lens 2516. The light then refracts onto the LCoS (liquidcrystal on silicon) chip 2510, where an image is formed. The light forthe image then reflects back through the field lens 2516 and ispolarized and reflected 90° through the polarizing beam splitter 2518.The light then leaves the microprojector for transmission to the opticaldisplay of the glasses.

FIG. 26 depicts an exemplary RGB LED module 2600. In this example, theLED is a 2×2 array with 1 red, 1 blue and 2 green die and the LED arrayhas 4 cathodes and a common anode. The maximum current may be 0.5 A perdie and the maximum voltage (≈4V) may be needed for the green and bluedie.

In embodiments, the system may utilize an optical system that is able togenerate a monochrome display to the wearer, which may provideadvantages to image clarity, image resolution, frame rate, and the like.For example, the frame rate may triple (over an RGB system) and this maybe useful in a night vision and the like situation where the camera isimaging the surroundings, where those images may be processed anddisplayed as content. The image may be brighter, such as be three timesbrighter if three LEDs are used, or provide a space savings with onlyone LED. If multiple LEDs are used, they may be the same color or theycould be different (RGB). The system may be a switchablemonochrome/color system where RGB is used but when the wearer wantsmonochrome they could either choose an individual LED or a number ofthem. All three LEDs may be used at the same time, as opposed tosequencing, to create white light. Using three LEDs without sequencingmay be like any other white light where the frame rate goes up by afactor of three. The “switching” between monochrome and color may bedone “manually” (e.g. a physical button, a GUI interface selection) orit may be done automatically depending on the application that isrunning. For instance, a wearer may go into a night vision mode or fogclearing mode, and the processing portion of the system automaticallydetermines that the eyepiece needs to go into a monochrome high refreshrate mode.

FIG. 3 depicts an embodiment of a horizontally disposed projector inuse. The projector 300 may be disposed in an arm portion of an eyepieceframe. The LED module 302, under processor control 304, may emit asingle color at a time in rapid sequence. The emitted light may traveldown a light tunnel 308 and through at least one homogenizing lenslet310 before encountering a polarizing beam splitter 312 and beingdeflected towards an LCoS display 314 where a full color image isdisplayed. The LCoS display may have a resolution of 1280×720 p. Theimage may then be reflected back up through the polarizing beamsplitter, reflected off a fold mirror 318 and travel through acollimator on its way out of the projector and into a waveguide. Theprojector may include a diffractive element to eliminate aberrations.

In an embodiment, the interactive head-mounted eyepiece includes anoptical assembly through which a user views a surrounding environmentand displayed content, wherein the optical assembly includes acorrective element that corrects the user's view of the surroundingenvironment, a freeform optical waveguide enabling internal reflections,and a coupling lens positioned to direct an image from an opticaldisplay, such as an LCoS display, to the optical waveguide. The eyepiecefurther includes one or more integrated processors for handling contentfor display to the user and an integrated image source, such as aprojector facility, for introducing the content to the optical assembly.In embodiments where the image source is a projector, the projectorfacility includes a light source and the optical display. Light from thelight source, such as an RGB module, is emitted under control of theprocessor and traverses a polarizing beam splitter where it is polarizedbefore being reflected off the optical display, such as the LCoS displayor LCD display in certain other embodiments, and into the opticalwaveguide. A surface of the polarizing beam splitter may reflect thecolor image from the optical display into the optical waveguide. The RGBLED module may emit light sequentially to form a color image that isreflected off the optical display. The corrective element may be asee-through correction lens that is attached to the optical waveguide toenable proper viewing of the surrounding environment whether the imagesource is on or off. This corrective element may be a wedge-shapedcorrection lens, and may be prescription, tinted, coated, or the like.The freeform optical waveguide, which may be described by a higher orderpolynomial, may include dual freeform surfaces that enable a curvatureand a sizing of the waveguide. The curvature and the sizing of thewaveguide enable its placement in a frame of the interactivehead-mounted eyepiece. This frame may be sized to fit a user's head in asimilar fashion to sunglasses or eyeglasses. Other elements of theoptical assembly of the eyepiece include a homogenizer through whichlight from the light source is propagated to ensure that the beam oflight is uniform and a collimator that improves the resolution of thelight entering the optical waveguide.

In embodiments, the prescription lens may be mounted on the inside ofthe eyepiece lens or on the outside. In some embodiments, theprescription power may be divided into prescription lenses mounted onthe outside and inside of the eyepiece lens. In embodiments, theprescription correction is provided by corrective optics that cling toeyepiece lens or a component of the optical assembly, such as thebeamsplitter, such as through surface tension. Suitable optics may beprovided by 3M's Press-On Optics, which are available at least as Prisms(a.k.a. Fresnel Prisms), Aspheric Minus Lenses, Aspheric Plus Lenses,and Bifocal Lenses. The corrective optics may be a user removable andreplaceable diopter correction facility adapted to be removably attachedin a position between the user's eye and the displayed content such thatthe diopter correction facility corrects the users eyesight with respectto the displayed content and the surrounding environment. The dioptercorrection facility may be adapted to mount to the optical assembly. Thediopter correction facility may be adapted to mount to the head-mountedeyepiece. The diopter correction facility may mount using a frictionfit. The diopter correction facility may mount using a magneticattachment facility. The user may select from a plurality of differentdiopter correction facilities depending on the user's eyesight.

Referring to FIG. 4, the image light, which may be polarized andcollimated, may optionally traverse a display coupling lens 412, whichmay or may not be the collimator itself or in addition to thecollimator, and enter the waveguide 414. In embodiments, the waveguide414 may be a freeform waveguide, where the surfaces of the waveguide aredescribed by a polynomial equation. The waveguide may be rectilinear.The waveguide 414 may include two reflective surfaces. When the imagelight enters the waveguide 414, it may strike a first surface with anangle of incidence greater than the critical angle above which totalinternal reflection (TIR) occurs. The image light may engage in TIRbounces between the first surface and a second facing surface,eventually reaching the active viewing area 418 of the composite lens.In an embodiment, light may engage in at least three TIR bounces. Sincethe waveguide 414 tapers to enable the TIR bounces to eventually exitthe waveguide, the thickness of the composite lens 420 may not beuniform. Distortion through the viewing area of the composite lens 420may be minimized by disposing a wedge-shaped correction lens 410 along alength of the freeform waveguide 414 in order to provide a uniformthickness across at least the viewing area of the lens 420. Thecorrection lens 410 may be a prescription lens, a tinted lens, apolarized lens, a ballistic lens, and the like, mounted on the inside oroutside of the eyepiece lense, or in some embodiments, mounted on boththe inside and outside of the eyepiece lens.

In some embodiments, while the optical waveguide may have a firstsurface and a second surface enabling total internal reflections of thelight entering the waveguide, the light may not actually enter thewaveguide at an internal angle of incidence that would result in totalinternal reflection. The eyepiece may include a mirrored surface on thefirst surface of the optical waveguide to reflect the displayed contenttowards the second surface of the optical waveguide. Thus, the mirroredsurface enables a total reflection of the light entering the opticalwaveguide or a reflection of at least a portion of the light enteringthe optical waveguide. In embodiments, the surface may be 100% mirroredor mirrored to a lower percentage. In some embodiments, in place of amirrored surface, an air gap between the waveguide and the correctiveelement may cause a reflection of the light that enters the waveguide atan angle of incidence that would not result in TIR.

In an embodiment, the eyepiece includes an integrated image source, suchas a projector, that introduces content for display to the opticalassembly from a side of the optical waveguide adjacent to an arm of theeyepiece. As opposed to prior art optical assemblies where imageinjection occurs from a top side of the optical waveguide, the presentdisclosure provides image injection to the waveguide from a side of thewaveguide. The displayed content aspect ratio is between approximatelysquare to approximately rectangular with the long axis approximatelyhorizontal. In embodiments, the displayed content aspect ratio is 16:9.In embodiments, achieving a rectangular aspect ratio for the displayedcontent where the long axis is approximately horizontal may be done viarotation of the injected image. In other embodiments, it may be done bystretching the image until it reaches the desired aspect ratio.

FIG. 5 depicts a design for a waveguide eyepiece showing sampledimensions. For example, in this design, the width of the coupling lens504 may be 13˜15 mm, with the optical display 502 optically coupled inseries. These elements may be disposed in an arm or redundantly in botharms of an eyepiece. Image light from the optical display 502 isprojected through the coupling lens 504 into the freeform waveguide 508.The thickness of the composite lens 520, including waveguide 508 andcorrection lens 510, may be 9 mm In this design, the waveguide 502enables an exit pupil diameter of 8 mm with an eye clearance of 20 mmThe resultant see-through view 512 may be about 60-70 mm The distancefrom the pupil to the image light path as it enters the waveguide 502(dimension a) may be about 50-60 mm, which can accommodate a large % ofhuman head breadths. In an embodiment, the field of view may be largerthan the pupil. In embodiments, the field of view may not fill the lens.It should be understood that these dimensions are for a particularillustrative embodiment and should not be construed as limiting. In anembodiment, the waveguide, snap-on optics, and/or the corrective lensmay comprise optical plastic. In other embodiments, the waveguidesnap-on optics, and/or the corrective lens may comprise glass, marginalglass, bulk glass, metallic glass, palladium-enriched glass, or othersuitable glass. In embodiments, the waveguide 508 and correction lens510 may be made from different materials selected to result in little tono chromatic aberrations. The materials may include a diffractiongrating, a holographic grating, and the like.

In embodiments such as that shown in FIG. 1, the projected image may bea stereo image when two projectors 108 are used for the left and rightimages. To enable stereo viewing, the projectors 108 may be disposed atan adjustable distance from one another that enables adjustment based onthe interpupillary distance for individual wearers of the eyepiece. Forexample, a single optical assembly may include two independentelectro-optic modules with individual adjustments for horizontal,vertical and tilt positioning. Alternatively, the optical assembly mayinclude only a single electro-optic module.

FIG. 6 depicts an embodiment of the eyepiece 600 with a see-through ortranslucent lens 602. A projected image 618 can be seen on the lens 602.In this embodiment, the image 618 that is being projected onto the lens602 happens to be an augmented reality version of the scene that thewearer is seeing, wherein tagged points of interest (POI) in the fieldof view are displayed to the wearer. The augmented reality version maybe enabled by a forward facing camera embedded in the eyepiece (notshown in FIG. 6) that images what the wearer is looking and identifiesthe location/POI. In one embodiment, the output of the camera or opticaltransmitter may be sent to the eyepiece controller or memory forstorage, for transmission to a remote location, or for viewing by theperson wearing the eyepiece or glasses. For example, the video outputmay be streamed to the virtual screen seen by the user. The video outputmay thus be used to help determine the user's location, or may be sentremotely to others to assist in helping to locate the location of thewearer, or for any other purpose. Other detection technologies, such asGPS, RFID, manual input, and the like, may be used to determine awearer's location. Using location or identification data, a database maybe accessed by the eyepiece for information that may be overlaid,projected or otherwise displayed with what is being seen. Augmentedreality applications and technology will be further described herein.

In FIG. 7, an embodiment of the eyepiece 700 is depicted with atranslucent lens 702 on which is being displayed streaming media (ane-mail application) and an incoming call notification 704. In thisembodiment, the media obscures a portion of the viewing area, however,it should be understood that the displayed image may be positionedanywhere in the field of view. In embodiments, the media may be made tobe more or less transparent.

In an embodiment, the eyepiece may receive input from any externalsource, such as an external converter box. The source may be depicted inthe lens of eyepiece. In an embodiment, when the external source is aphone, the eyepiece may use the phone's location capabilities to displaylocation-based augmented reality, including marker overlay frommarker-based AR applications. In embodiments, a VNC client running onthe eyepiece's processor or an associated device may be used to connectto and control a computer, where the computer's display is seen in theeyepiece by the wearer. In an embodiment, content from any source may bestreamed to the eyepiece, such as a display from a panoramic camerariding atop a vehicle, a user interface for a device, imagery from adrone or helicopter, and the like. For example, a gun-mounted camera mayenable shooting a target not in direct line of sight when the camerafeed is directed to the eyepiece.

The lenses may be chromic, such as photochromic or electrochromic. Theelectrochromic lens may include integral chromic material or a chromiccoating which changes the opacity of at least a portion of the lens inresponse to a burst of charge applied by the processor across thechromic material. For example, and referring to FIG. 9, a chromicportion 902 of the lens 904 is shown darkened, such as for providinggreater viewability by the wearer of the eyepiece when that portion isshowing displayed content to the wearer. In embodiments, there may be aplurality of chromic areas on the lens that may be controlledindependently, such as large portions of the lens, sub-portions of theprojected area, programmable areas of the lens and/or projected area,controlled to the pixel level, and the like. Activation of the chromicmaterial may be controlled via the control techniques further describedherein or automatically enabled with certain applications (e.g. astreaming video application, a sun tracking application, an ambientbrightness sensor, a camera tracking brightness in the field of view) orin response to a frame-embedded UV sensor. In embodiments, anelectrochromic layer may be located between optical elements and/or onthe surface of an optical element on the eyepiece, such as on acorrective lens, on a ballistic lens, and the like. In an example, theelectrochromic layer may consist of a stack, such as an Indium Tin Oxide(ITO) coated PET/PC film with two layers of electrochromic (EC) between,which may eliminate another layer of PET/PC, thereby reducingreflections (e.g. a layer stack may comprise aPET/PC-EC-PET/PC-EC-PET/PC). In embodiments, the electricallycontrollable optical layer may be provided as a liquid crystal basedsolution with a binary state of tint. In other embodiments, multiplelayers of liquid crystal or an alternative e-tint forming the opticallayer may be used to provide variable tint such that certain layers orsegments of the optical layer may be turned on or off in stages.Electrochromic layers may be used generically for any of theelectrically controlled transparencies in the eyepiece, including SPD,LCD, electrowetting, and the like.

In embodiments, the lens may have an angular sensitive coating whichenables transmitting light-waves with low incident angles and reflectinglight, such as s-polarized light, with high incident angles. The chromiccoating may be controlled in portions or in its entirety, such as by thecontrol technologies described herein. The lenses may be variablecontrast and the contrast may be under the control of a push button orany other control technique described herein. In embodiments, the usermay wear the interactive head-mounted eyepiece, where the eyepieceincludes an optical assembly through which the user views a surroundingenvironment and displayed content. The optical assembly may include acorrective element that corrects the user's view of the surroundingenvironment, an integrated processor for handling content for display tothe user, and an integrated image source for introducing the content tothe optical assembly. The optical assembly may include an electrochromiclayer that provides a display characteristic adjustment that isdependent on displayed content requirements and surroundingenvironmental conditions. In embodiments, the display characteristic maybe brightness, contrast, and the like. The surrounding environmentalcondition may be a level of brightness that without the displaycharacteristic adjustment would make the displayed content difficult tovisualize by the wearer of the eyepiece, where the displaycharacteristic adjustment may be applied to an area of the opticalassembly where content is being displayed.

In embodiments, the eyepiece may have brightness, contrast, spatial,resolution, and the like control over the eyepiece projected area, suchas to alter and improve the user's view of the projected content againsta bright or dark surrounding environment. For example, a user may beusing the eyepiece under bright daylight conditions, and in order forthe user to clearly see the displayed content the display area my needto be altered in brightness and/or contrast. Alternatively, the viewingarea surrounding the display area may be altered. In addition, the areaaltered, whether within the display area or not, may be spatiallyoriented or controlled per the application being implemented. Forinstance, only a small portion of the display area may need to bealtered, such as when that portion of the display area deviates fromsome determined or predetermined contrast ratio between the displayportion of the display area and the surrounding environment. Inembodiments, portions of the lens may be altered in brightness,contrast, spatial extent, resolution, and the like, such as fixed toinclude the entire display area, adjusted to only a portion of the lens,adaptable and dynamic to changes in lighting conditions of thesurrounding environment and/or the brightness-contrast of the displayedcontent, and the like. Spatial extent (e.g. the area affected by thealteration) and resolution (e.g. display optical resolution) may varyover different portions of the lens, including high resolution segments,low resolution segments, single pixel segments, and the like, wherediffering segments may be combined to achieve the viewing objectives ofthe application(s) being executed. In embodiments, technologies forimplementing alterations of brightness, contrast, spatial extent,resolution, and the like, may include electrochromic materials, LCDtechnologies, embedded beads in the optics, flexible displays,suspension particle device (SPD) technologies, colloid technologies, andthe like.

In embodiments, there may be various modes of activation of theelectrochromic layer. For example, the user may enter sunglass modewhere the composite lenses appear only somewhat darkened or the user mayenter “Blackout” mode, where the composite lenses appear completelyblackened.

In an example of a technology that may be employed in implementing thealterations of brightness, contrast, spatial extent, resolution, and thelike, may be electrochromic materials, films, inks, and the like.Electrochromism is the phenomenon displayed by some materials ofreversibly changing appearance when electric charge is applied. Varioustypes of materials and structures can be used to constructelectrochromic devices, depending on the specific applications. Forinstance, electrochromic materials include tungsten oxide (WO₃), whichis the main chemical used in the production of electrochromic windows orsmart glass. In embodiments, electrochromic coatings may be used on thelens of the eyepiece in implementing alterations. In another example,electrochromic displays may be used in implementing ‘electronic paper’,which is designed to mimic the appearance of ordinary paper, where theelectronic paper displays reflected light like ordinary paper. Inembodiments, electrochromism may be implemented in a wide variety ofapplications and materials, including gyricon (consisting ofpolyethylene spheres embedded in a transparent silicone sheet, with eachsphere suspended in a bubble of oil so that they can rotate freely),electro-phoretic displays (forming images by rearranging charged pigmentparticles using an applied electric field), E-Ink technology,electro-wetting, electro-fluidic, interferometric modulator, organictransistors embedded into flexible substrates, nano-chromics displays(NCD), and the like.

In another example of a technology that may be employed in implementingthe alterations of brightness, contrast, spatial extent, resolution, andthe like, may be suspended particle devices (SPD). When a small voltageis applied to an SPD film, its microscopic particles, which in theirstable state are randomly dispersed, become aligned and allow light topass through. The response may be immediate, uniform, and with stablecolor throughout the film. Adjustment of the voltage may allow users tocontrol the amount of light, glare and heat passing through. Thesystem's response may range from a dark blue appearance, with up to fullblockage of light in its off state, to clear in its on state. Inembodiments, SPD technology may be an emulsion applied on a plasticsubstrate creating the active film. This plastic film may be laminated(as a single glass pane), suspended between two sheets of glass, plasticor other transparent materials, and the like.

Referring to FIGS. 8A-C, in certain embodiments, the electro-optics maybe mounted in a monocular or binocular flip-up/flip-down arrangement intwo parts: 1) electro-optics; and 2) correction lens. FIG. 8A depicts atwo part eyepiece where the electro-optics are contained within a module802 that may be electrically connected to the eyepiece 804 via anelectrical connector 810, such as a plug, pin, socket, wiring, and thelike. In this arrangement, the lens 818 in the frame 814 may be acorrection lens entirely. The interpupillary distance (IPD) between thetwo halves of the electro-optic module 802 may be adjusted at the bridge808 to accommodate various IPDs. Similarly, the placement of the display812 may be adjusted via the bridge 808. FIG. 8B depicts the binocularelectro-optics module 802 where one half is flipped up and the otherhalf is flipped down. The nose bridge may be fully adjustable andelastomeric. This enables 3-point mounting on nose bridge and ears witha head strap to assure the stability of images in the user's eyes,unlike the instability of helmet-mounted optics, that shift on thescalp. Referring to FIG. 8C, the lens 818 may be ANSI-compliant,hard-coat scratch-resistant polycarbonate ballistic lenses, may bechromic, may have an angular sensitive coating, may include aUV-sensitive material, and the like. In this arrangement, theelectro-optics module may include a CMOS-based VIS/NIR/SWIR blacksilicon sensor for night vision capability. The electro-optics module802 may feature quick disconnect capability for user flexibility, fieldreplacement and upgrade. The electro-optics module 802 may feature anintegrated power dock.

As in FIG. 79, the flip-up/flip-down lens 7910 may include a light block7908. Removable, elastomeric night adapters/light dams/light blocks 7908may be used to shield the flip-up/flip-down lens 7910, such as for nightoperations. The exploded top view of the eyepiece also depicts aheadstrap 7900, frame 7904, and adjustable nose bridge 7902. FIG. 80depicts an exploded view of the electro-optic assembly in a front (A)and side angle (B) view. A holder 8012 holds the see-through optic withcorrective lens 7910. An O-ring 8020 and screw 8022 secures the holderto the shaft 8024. A spring 8028 provides a spring-loaded connectionbetween the holder 8012 and shaft 8024. The shaft 8024 connects to theattachment bracket 8014, which secures to the eyepiece using thethumbscrew 8018. The shaft 8024 serves as a pivot and an IPD adjustmenttool using the IPD adjustment knob 8030. As seen in FIG. 81, the knob8030 rotates along adjustment threads 8134. The shaft 8024 also featurestwo set screw grooves 8132.

In embodiments, a photochromic layer may be included as part of theoptics of the eyepiece. Photochromism is the reversible transformationof a chemical species between two forms by the absorption ofelectromagnetic radiation, where the two forms have different absorptionspectra, such as a reversible change of color, darkness, and the like,upon exposure to a given frequency of light. In an example, aphotochromic layer may be included between the waveguide and correctiveoptics of the eyepiece, on the outside of the corrective optic, and thelike. In embodiments, a photochromic layer (such as used as a darkeninglayer) may be activated with a UV diode, or other photochromicresponsive wavelength known in the art. In the case of the photochromiclayer being activated with UV light, the eyepiece optics may alsoinclude a UV coating outside the photochromic layer to prevent UV lightfrom the Sun from accidentally activating it.

Photochromics are presently fast to change from light to dark and slowto change from dark to light. This due to the molecular changes that areinvolved with the photochromic material changing from clear to dark.Photochromic molecules are vibrating back to clear after the UV light,such as UV light from the sun, is removed. By increasing the vibrationof the molecules, such as by exposure to heat, the optic will clearquicker. The speed at which the photochromic layer goes from dark tolight may be temperature-dependent. Rapid changing from dark to light isparticularly important for military applications where users ofsunglasses often go from a bright outside environment to a dark insideenvironment and it is important to be able to see quickly in the insideenvironment.

This disclosure provides a photochromic film device with an attachedheater that is used to accelerate the transition from dark to clear inthe photochromic material. This method relies on the relationshipbetween the speed of transition of photochromic materials from dark toclear wherein the transition is faster at higher temperatures. To enablethe heater to increase the temperature of the photochromic materialrapidly, the photochromic material is provided as a thin layer with athin heater. By keeping the thermal mass of the photochromic film devicelow per unit area, the heater only has to provide a small amount of heatto rapidly produce a large temperature change in the photochromicmaterial. Since the photochromic material only needs to be at a highertemperature during the transition from dark to clear, the heater onlyneeds to be used for short periods of time so the power requirement islow.

The heater may be a thin and transparent heater element, such as an ITOheater or any other transparent and electrically conductive filmmaterial. When a user needs the eyepiece to go clear quickly, the usermay activate the heater element by any of the control techniquesdiscussed herein.

In an embodiment, the heater element may be used to calibrate thephotochromic element to compensate for cold ambient conditions when thelenses might go dark on their own.

In another embodiment, a thin coat of photochromic material may bedeposited on a thick substrate with the heater element layered on top.For example, the cover sunglass lens may comprise an acceleratedphotochromic solution and still have a separate electrochromic patchover the display area that may optionally be controlled with or withoutUV light.

FIG. 94A depicts a photochromic film device with a serpentine heaterpattern and FIG. 94B depicts a side view of a photochromic film devicewherein the device is a lens for sunglasses. The photochromic filmdevice is shown above and not contacting a protective cover lens toreduce the thermal mass of the device.

U.S. Pat. No. 3,152,215 describes a heater layer combined with aphotochromic layer to heat the photochromic material for the purpose ofreducing the time to transition from dark to clear. However, thephotochromic layer is positioned in a wedge which would greatly increasethe thermal mass of the device and thereby decrease the rate that theheater could change the temperature of the photochromic material oralternately greatly increase the power required to change thetemperature of the photochromic material.

This disclosure includes the use of a thin carrier layer that thephotochromic material is applied to. The carrier layer can be glass orplastic. The photochromic material can be applied by vacuum coating, bydipping or by thermal diffusion into the carrier layer as is well knownin the art. The thickness of the carrier layer can be 150 microns orless. The selection of the thickness of the carrier layer is selectedbased on the desired darkness of the photochromic film device in thedark state and the desired speed of transition between the dark stateand the clear state. Thicker carrier layers can be darker in the darkstate while being slower to heat to an elevated temperature due tohaving more thermal mass. Conversely, thinner carrier layers can be lessdark in the dark state while being faster to heat to an elevatedtemperature due to having less thermal mass.

The protective layer shown in FIG. 94 is separated from the photochromicfilm device to keep the thermal mass of the photochromic film devicelow. In this way, the protective layer can be made thicker to providehigher impact strength. The protective layer can be glass or plastic,for example the protective layer can be polycarbonate.

The heater can be a transparent conductor that is patterned into aconductive path that is relatively uniform so that the heat generatedover the length of the patterned heater is relatively uniform. Anexample of a transparent conductor that can be patterned is titaniumdioxide. A larger area is provided at the ends of the heater pattern forelectrical contacts such as is shown in FIG. 94.

As noted in the discussion for FIG. 8A-C, the augmented reality glassesmay include a lens 818 for each eye of the wearer. The lenses 818 may bemade to fit readily into the frame 814, so that each lens may betailored for the person for whom the glasses are intended. Thus, thelenses may be corrective lenses, and may also be tinted for use assunglasses, or have other qualities suitable for the intendedenvironment. Thus, the lenses may be tinted yellow, dark or othersuitable color, or may be photochromic, so that the transparency of thelens decreases when exposed to brighter light. In one embodiment, thelenses may also be designed for snap fitting into or onto the frames,i.e., snap on lenses are one embodiment. For example, the lenses may bemade from high-quality Schott optical glass and may include a polarizingfilter.

Of course, the lenses need not be corrective lenses; they may simplyserve as sunglasses or as protection for the optical system within theframe. In non-flip up/flip down arrangements, it goes without sayingthat the outer lenses are important for helping to protect the ratherexpensive waveguides, viewing systems and electronics within theaugmented reality glasses. At a minimum, the outer lenses offerprotection from scratching by the environment of the user, whether sand,brambles, thorns and the like, in one environment, and flying debris,bullets and shrapnel, in another environment. In addition, the outerlenses may be decorative, acting to change a look of the composite lens,perhaps to appeal to the individuality or fashion sense of a user. Theouter lenses may also help one individual user to distinguish his or herglasses from others, for example, when many users are gathered together.

It is desirable that the lenses be suitable for impact, such as aballistic impact. Accordingly, in one embodiment, the lenses and theframes meet ANSI Standard Z87.1-2010 for ballistic resistance. In oneembodiment, the lenses also meet ballistic standard CE EN166B. Inanother embodiment, for military uses, the lenses and frames may meetthe standards of MIL-PRF-31013, standards 3.5.1.1 or 4.4.1.1. Each ofthese standards has slightly different requirements for ballisticresistance and each is intended to protect the eyes of the user fromimpact by high-speed projectiles or debris. While no particular materialis specified, polycarbonate, such as certain Lexan® grades, usually issufficient to pass tests specified in the appropriate standard.

In one embodiment, as shown in FIG. 8D, the lenses snap in from theoutside of the frame, not the inside, for better impact resistance,since any impact is expected from the outside of the augmented realityeyeglasses. In this embodiment, replaceable lens 819 has a plurality ofsnap-fit arms 819 a which fit into recesses 820 a of frame 820. Theengagement angle 819 b of the arm is greater than 90°, while theengagement angle 820 b of the recess is also greater than 90°. Makingthe angles greater than right angles has the practical effect ofallowing removal of lens 819 from the frame 820. The lens 819 may needto be removed if the person's vision has changed or if a different lensis desired for any reason. The design of the snap fit is such that thereis a slight compression or bearing load between the lens and the frame.That is, the lens may be held firmly within the frame, such as by aslight interference fit of the lens within the frame.

The cantilever snap fit of FIG. 8D is not the only possible way toremovably snap-fit the lenses and the frame. For example, an annularsnap fit may be used, in which a continuous sealing lip of the frameengages an enlarged edge of the lens, which then snap-fits into the lip,or possibly over the lip. Such a snap fit is typically used to join acap to an ink pen. This configuration may have an advantage of asturdier joint with fewer chances for admission of very small dust anddirt particles. Possible disadvantages include the fairly tighttolerances required around the entire periphery of both the lens andframe, and the requirement for dimensional integrity in all threedimensions over time.

It is also possible to use an even simpler interface, which may still beconsidered a snap-fit. A groove may be molded into an outer surface ofthe frame, with the lens having a protruding surface, which may beconsidered a tongue that fits into the groove. If the groove issemi-cylindrical, such as from about 270° to about 300°, the tongue willsnap into the groove and be firmly retained, with removal still possiblethrough the gap that remains in the groove. In this embodiment, shown inFIG. 8E, a lens or replacement lens or cover 826 with a tongue 828 maybe inserted into a groove 827 in a frame 825, even though the lens orcover is not snap-fit into the frame. Because the fit is a close one, itwill act as a snap-fit and securely retain the lens in the frame.

In another embodiment, the frame may be made in two pieces, such as alower portion and an upper portion, with a conventionaltongue-and-groove fit. In another embodiment, this design may also usestandard fasteners to ensure a tight grip of the lens by the frame. Thedesign should not require disassembly of anything on the inside of theframe. Thus, the snap-on or other lens or cover should be assembled ontothe frame, or removed from the frame, without having to go inside theframe. As noted in other parts of this disclosure, the augmented realityglasses have many component parts. Some of the assemblies andsubassemblies may require careful alignment. Moving and jarring theseassemblies may be detrimental to their function, as will moving andjarring the frame and the outer or snap-on lens or cover.

In embodiments, the flip-up/flip-down arrangement enables a modulardesign for the eyepiece. For example, not only can the eyepiece beequipped with a monocular or binocular module 802, but the lens 818 mayalso be swapped. In embodiments, additional features may be includedwith the module 802, either associated with one or both displays 812.Referring to FIG. 8F, either monocular or binocular versions of themodule 802 may be display only 852 (monocular), 854 (binocular) or maybe equipped with a forward-looking camera 858 (monocular), and 860 & 862(binocular). In some embodiments, the module may have additionalintegrated electronics, such as a GPS, a laser range finder, and thelike. In the embodiment 862 enabling urban leader tactical response,awareness & visualization, also known as ‘Ultra-Vis’, a binocularelectro-optic module 862 is equipped with stereo forward-looking cameras870, GPS, and a laser range finder 868. These features may enable theUltra-Vis embodiment to have panoramic night vision, and panoramic nightvision with laser range finder and geo location.

In an embodiment, the electro-optics characteristics may be, but notlimited to, as follows:

Optic Characteristics Value WAVEGUIDE virtual display field of view~25-30 degrees (equivalent to the FOV (Diagonal) of a 24″ monitor viewedat 1 m distance) see-through field of view more than 80 degrees eyeclearance more than 18 mm Material zeonex optical plastic weight approx15 grams Wave Guide dimensions 60 × 30 × 10 mm (or 9) Size 15.5 mm(diagonal) Material PMMA (optical plastics) FOV 53.5° (diagonal) Activedisplay area 12.7 mm × 9.0 mm Resolution 800 × 600 pixels VIRTUALIMAGING SYSTEM Type Folded FFS prism Effective focal length 15 mm Exitpupil diameter 8 mm Eye relief 18.25 mm F# 1.875 Number of free formsurfaces 2-3 AUGMENTED VIEWING SYSTEM Type Free form Lens Number of freeform surfaces 2 Other Parameters Wavelength 656.3-486.1 nm Field of view45°H × 32°V Vignetting 0.15 for the top and bottom fields Distortion<12% at the maximum field Image quality MTF > 10% at 30 lp/mm

In an embodiment, the Projector Characteristics may be as follows:

Projector Characteristics Value Brightness Adjustable, .25-2 LumensVoltage 3.6 VDC Illumination Red, Green and Blue LEDs Display SVGA 800 ×600 dpi Syndiant LCOS Display Power Consumption Adjustable, 50 to 250 mwTarget MPE Dimensions Approximately 24 mm × 12 mm × 6 mm FocusAdjustable Optics Housing 6061-T6 Aluminum and Glass-filled ABS/PCWeight 5 gms RGB Engine Adjustable Color Output ARCHITECTURE 2 × 1 GHZprocessor cores 633 MHZ DSPs 30M polygons/sec DC graphics acceleratorIMAGE CORRECTION real-time sensing image enhancement noise reductionkeystone correction perspective correction

In another embodiment, an augmented reality eyepiece may includeelectrically-controlled lenses as part of the microprojector or as partof the optics between the microprojector and the waveguide. FIG. 21depicts an embodiment with such liquid lenses 2152.

The glasses may also include at least one camera or optical sensor 2130that may furnish an image or images for viewing by the user. The imagesare formed by a microprojector 2114 on each side of the glasses forconveyance to the waveguide 2108 on that side. In one embodiment, anadditional optical element, a variable focus lens 2152 may also befurnished. The lens may be electrically adjustable by the user so thatthe image seen in the waveguides 2108 are focused for the user. Inembodiments, the camera may be a multi-lens camera, such as an ‘arraycamera’, where the eyepiece processor may combine the data from themultiple lenses and multiple viewpoints of the lenses to build a singlehigh-quality image. This technology may be referred to as computationalimaging, since software is used to process the image. Computationalimaging may provide image-processing advantages, such as allowingprocessing of the composite image as a function of individual lensimages. For example, since each lens may provide it's own image, theprocessor may provide image processing to create images with specialfocusing, such as foveal imaging, where the focus from one of the lensimages is clear, higher resolution, and the like, and where the rest ofthe image is defocused, lower resolution, and the like. The processormay also select portions of the composite image to store in memory,while deleting the rest, such as when memory storage is limited and onlyportions of the composite image are critical to save. In embodiments,use of the array camera may provide the ability to alter the focus of animage after the image has been taken. In addition to the imagingadvantages of an array camera, the array camera may provide a thinnermechanical profile than a traditional single-lens assembly, thus makingit easier to integrate into the eyepiece.

Variable lenses may include the so-called liquid lenses furnished byVarioptic, S.A., Lyons, France, or by LensVector, Inc., Mountain View,Calif., U.S.A. Such lenses may include a central portion with twoimmiscible liquids. Typically, in these lenses, the path of lightthrough the lens, i.e., the focal length of the lens is altered orfocused by applying an electric potential between electrodes immersed inthe liquids. At least one of the liquids is affected by the resultingelectric or magnetic field potential. Thus, electrowetting may occur, asdescribed in U.S. Pat. Appl. Publ. 2010/0007807, assigned to LensVector,Inc. Other techniques are described in LensVector Pat. Appl. Pubis.2009/021331 and 2009/0316097. All three of these disclosures areincorporated herein by reference, as though each page and figures wereset forth verbatim herein.

Other patent documents from Varioptic, S.A., describe other devices andtechniques for a variable focus lens, which may also work through anelectrowetting phenomenon. These documents include U.S. Pats. No.7,245,440 and 7,894,440 and U.S. Pat. Appl. Pubis. 2010/0177386 and2010/0295987, each of which is also incorporated herein by reference, asthough each page and figures were set forth verbatim herein. In thesedocuments, the two liquids typically have different indices ofrefraction and different electrical conductivities, e.g., one liquid isconductive, such as an aqueous liquid, and the other liquid isinsulating, such as an oily liquid. Applying an electric potential maychange the thickness of the lens and does change the path of lightthrough the lens, thus changing the focal length of the lens.

The electrically-adjustable lenses may be controlled by the controls ofthe glasses. In one embodiment, a focus adjustment is made by calling upa menu from the controls and adjusting the focus of the lens. The lensesmay be controlled separately or may be controlled together. Theadjustment is made by physically turning a control knob, by indicatingwith a gesture, or by voice command. In another embodiment, theaugmented reality glasses may also include a rangefinder, and focus ofthe electrically-adjustable lenses may be controlled automatically bypointing the rangefinder, such as a laser rangefinder, to a target orobject a desired distance away from the user.

As shown in U.S. Pat. No. 7,894,440, discussed above, the variablelenses may also be applied to the outer lenses of the augmented realityglasses or eyepiece. In one embodiment, the lenses may simply take theplace of a corrective lens. The variable lenses with theirelectric-adjustable control may be used instead of or in addition to theimage source- or projector-mounted lenses. The corrective lens insertsprovide corrective optics for the user's environment, the outside world,whether the waveguide displays are active or not.

It is important to stabilize the images presented to the wearer of theaugmented reality glasses or eyepiece(s), that is, the images seen inthe waveguide. The view or images presented travel from one or twodigital cameras or sensors mounted on the eyepiece, to digitalcircuitry, where the images are processed and, if desired, stored asdigital data before they appear in the display of the glasses. In anyevent, and as discussed above, the digital data is then used to form animage, such as by using an LCOS display and a series of RGB lightemitting diodes. The light images are processed using a series oflenses, a polarizing beam splitter, an electrically-powered liquidcorrective lens and at least one transition lens from the projector tothe waveguide.

The process of gathering and presenting images includes severalmechanical and optical linkages between components of the augmentedreality glasses. It seems clear, therefore, that some form ofstabilization will be required. This may include optical stabilizationof the most immediate cause, the camera itself, since it is mounted on amobile platform, the glasses, which themselves are movably mounted on amobile user. Accordingly, camera stabilization or correction may berequired. In addition, at least some stabilization or correction shouldbe used for the liquid variable lens. Ideally, a stabilization circuitat that point could correct not only for the liquid lens, but also forany aberration and vibration from many parts of the circuit upstreamfrom the liquid lens, including the image source. One advantage of thepresent system is that many commercial off-the-shelf cameras are veryadvanced and typically have at least one image-stabilization feature oroption. Thus, there may be many embodiments of the present disclosure,each with a same or a different method of stabilizing an image or a veryfast stream of images, as discussed below. The term opticalstabilization is typically used herein with the meaning of physicallystabilizing the camera, camera platform, or other physical object, whileimage stabilization refers to data manipulation and processing.

One technique of image stabilization is performed on digital images asthey are formed. This technique may use pixels outside the border of thevisible frame as a buffer for the undesired motion. Alternatively, thetechnique may use another relatively steady area or basis in succeedingframes. This technique is applicable to video cameras, shifting theelectronic image from frame to frame of the video in a manner sufficientto counteract the motion. This technique does not depend on sensors anddirectly stabilizes the images by reducing vibrations and otherdistracting motion from the moving camera. In some techniques, the speedof the images may be slowed in order to add the stabilization process tothe remainder of the digital process, and requiring more time per image.These techniques may use a global motion vector calculated fromframe-to-frame motion differences to determine the direction of thestabilization.

Optical stabilization for images uses a gravity- orelectronically-driven mechanism to move or adjust an optical element orimaging sensor such that it counteracts the ambient vibrations. Anotherway to optically stabilize the displayed content is to providegyroscopic correction or sensing of the platform housing the augmentedreality glasses, e.g., the user. As noted above, the sensors availableand used on the augmented reality glasses or eyepiece include MEMSgyroscopic sensors. These sensors capture movement and motion in threedimensions in very small increments and can be used as feedback tocorrect the images sent from the camera in real time. It is clear thatat least a large part of the undesired and undesirable movement probablyis caused by movement of the user and the camera itself. These largermovements may include gross movements of the user, e.g., walking orrunning, riding in a vehicle. Smaller vibrations may also result withinthe augmented reality eyeglasses, that is, vibrations in the componentsin the electrical and mechanical linkages that form the path from thecamera (input) to the image in the waveguide (output). These grossmovements may be more important to correct or to account for, ratherthan, for instance, independent and small movements in the linkages ofcomponents downstream from the projector. In embodiments, the gyroscopicstabilization may stabilize the image when it is subject to a periodicmotion. For such periodic motion, the gyroscope may determine theperiodicity of the user's motion and transmit the information to aprocessor to correct for the placement of content in the user's view.The gyroscope may utilize a rolling average of two or three or morecycles of periodic motion in determining the periodicity. Other sensorsmay also be used to stabilize the image or correctly place the image inthe user's field of view, such as an accelerometer, a position sensor, adistance sensor, a rangefinder, a biological sensor, a geodetic sensor,an optical sensor, a video sensor, a camera, an infrared sensor, a lightsensor, a photocell sensor, or an RF sensor. When a sensor detects userhead or eye movement, the sensor provides an output to a processor whichmay determine the direction, speed, amount, and rate of the user's heador eye movement. The processor may convert this information into asuitable data structure for further processing by the processorcontrolling the optical assembly (which may be the same processor). Thedata structure may be one or more vector quantities. For example, thedirection of the vector may define the orientation of the movement, andthe length of the vector may define the rate of the movement. Using theprocessed sensor output, the display of content is adjusted accordingly.

Motion sensing may thus be used to sense the motion and correct for it,as in optical stabilization, or to sense the motion and then correct theimages that are being taken and processed, as in image stabilization. Anapparatus for sensing motion and correcting the images or the data isdepicted in FIG. 34A. In this apparatus, one or more kinds of motionsensors may be used, including accelerometers, angular position sensorsor gyroscopes, such as MEMS gyroscopes. Data from the sensors is fedback to the appropriate sensor interfaces, such as analog to digitalconverters (ADCs) or other suitable interface, such as digital signalprocessors (DSPs). A microprocessor then processes this information, asdiscussed above, and sends image-stabilized frames to the display driverand then to the see-through display or waveguide discussed above. In oneembodiment, the display begins with the RGB display in themicroprojector of the augmented reality eyepiece.

In another embodiment, a video sensor or augmented reality glasses, orother device with a video sensor may be mounted on a vehicle. In thisembodiment, the video stream may be communicated through atelecommunication capability or an Internet capability to personnel inthe vehicle. One application could be sightseeing or touring of an area.Another embodiment could be exploring or reconnaissance, or evenpatrolling, of an area. In these embodiments, gyroscopic stabilizationof the image sensor would be helpful, rather than applying a gyroscopiccorrection to the images or digital data representing the images. Anembodiment of this technique is depicted in FIG. 34B. In this technique,a camera or image sensor 3407 is mounted on a vehicle 3401. One or moremotion sensors 3406, such as gyroscopes, are mounted in the cameraassembly 3405. A stabilizing platform 3403 receives information from themotion sensors and stabilizes the camera assembly 3405, so that jitterand wobble are minimized while the camera operates. This is true opticalstabilization. Alternatively, the motion sensors or gyroscopes may bemounted on or within the stabilizing platform itself. This techniquewould actually provide optical stabilization, stabilizing the camera orimage sensor, in contrast to digital stabilization, correcting the imageafterwards by computer processing of the data taken by the camera.

In one technique, the key to optical stabilization is to apply thestabilization or correction before an image sensor converts the imageinto digital information. In one technique, feedback from sensors, suchas gyroscopes or angular velocity sensors, is encoded and sent to anactuator that moves the image sensor, much as an autofocus mechanismadjusts a focus of a lens. The image sensor is moved in such a way as tomaintain the projection of the image onto the image plane, which is afunction of the focal length of the lens being used. Autoranging andfocal length information, perhaps from a range finder of the interactivehead-mounted eyepiece, may be acquired through the lens itself. Inanother technique, angular velocity sensors, sometimes also calledgyroscopic sensors, can be used to detect, respectively, horizontal andvertical movements. The motion detected may then be fed back toelectromagnets to move a floating lens of the camera. This opticalstabilization technique, however, would have to be applied to each lenscontemplated, making the result rather expensive.

Stabilization of the liquid lens is discussed in U.S. Pat. Appl. Publ.2010/0295987, assigned to Varioptic, S.A., Lyon, France. In theory,control of a liquid lens is relatively simple, since there is only onevariable to control: the level of voltage applied to the electrodes inthe conducting and non-conducting liquids of the lens, using, forexamples, the lens housing and the cap as electrodes. Applying a voltagecauses a change or tilt in the liquid-liquid interface via theelectrowetting effect. This change or tilt adjusts the focus or outputof the lens. In its most basic terms, a control scheme with feedbackwould then apply a voltage and determine the effect of the appliedvoltage on the result, i.e., a focus or an astigmatism of the image. Thevoltages may be applied in patterns, for example, equal and opposite +and − voltages, both positive voltages of differing magnitude, bothnegative voltages of differing magnitude, and so forth. Such lenses areknown as electrically variable optic lenses or electro-optic lenses.

Voltages may be applied to the electrodes in patterns for a short periodof time and a check on the focus or astigmatism made. The check may bemade, for instance, by an image sensor. In addition, sensors on thecamera or in this case the lens, may detect motion of the camera orlens. Motion sensors would include accelerometers, gyroscopes, angularvelocity sensors or piezoelectric sensors mounted on the liquid lens ora portion of the optic train very near the liquid lens. In oneembodiment, a table, such as a calibration table, is then constructed ofvoltages applied and the degree of correction or voltages needed forgiven levels of movement. More sophistication may also be added, forexample, by using segmented electrodes in different portions of theliquid so that four voltages may be applied rather than two. Of course,if four electrodes are used, four voltages may be applied, in many morepatterns than with only two electrodes. These patterns may include equaland opposite positive and negative voltages to opposite segments, and soforth. An example is depicted in FIG. 34C. Four electrodes 3409 aremounted within a liquid lens housing (not shown). Two electrodes aremounted in or near the non-conducting liquid and two are mounted in ornear the conducting liquid. Each electrode is independent in terms ofthe possible voltage that may be applied.

Look-up or calibration tables may be constructed and placed in thememory of the augmented reality glasses. In use, the accelerometer orother motion sensor will sense the motion of the glasses, i.e., thecamera on the glasses or the lens itself. A motion sensor such as anaccelerometer will sense in particular, small vibration-type motionsthat interfere with smooth delivery of images to the waveguide. In oneembodiment, the image stabilization techniques described here can beapplied to the electrically-controllable liquid lens so that the imagefrom the projector is corrected immediately. This will stabilize theoutput of the projector, at least partially correcting for the vibrationand movement of the augmented reality eyepiece, as well as at least somemovement by the user. There may also be a manual control for adjustingthe gain or other parameter of the corrections. Note that this techniquemay also be used to correct for near-sightedness or far-sightedness ofthe individual user, in addition to the focus adjustment alreadyprovided by the image sensor controls and discussed as part of theadjustable-focus projector.

Another variable focus element uses tunable liquid crystal cells tofocus an image. These are disclosed, for example, in U.S. Pat. Appl.Publ. Nos. 2009/0213321, 2009/0316097 and 2010/0007807, which are herebyincorporated by reference in their entirety and relied on. In thismethod, a liquid crystal material is contained within a transparentcell, preferably with a matching index of refraction. The cell includestransparent electrodes, such as those made from indium tin oxide (ITO).Using one spiral-shaped electrode, and a second spiral-shaped electrodeor a planar electrode, a spatially non-uniform magnetic field isapplied. Electrodes of other shapes may be used. The shape of themagnetic field determines the rotation of molecules in the liquidcrystal cell to achieve a change in refractive index and thus a focus ofthe lens. The liquid crystals can thus be electromagneticallymanipulated to change their index of refraction, making the tunableliquid crystal cell act as a lens.

In a first embodiment, a tunable liquid crystal cell 3420 is depicted inFIG. 34D. The cell includes an inner layer of liquid crystal 3421 andthin layers 3423 of orienting material such as polyimide. This materialhelps to orient the liquid crystals in a preferred direction.Transparent electrodes 3425 are on each side of the orienting material.An electrode may be planar, or may be spiral shaped as shown on theright in FIG. 34D. Transparent glass substrates 3427 contain thematerials within the cell. The electrodes are formed so that they willlend shape to the magnetic field. As noted, a spiral shaped electrode onone or both sides, such that the two are not symmetrical, is used in oneembodiment. A second embodiment is depicted in FIG. 34E. Tunable liquidcrystal cell 3430 includes central liquid crystal material 3431,transparent glass substrate walls 3433, and transparent electrodes.Bottom electrode 3435 is planar, while top electrode 3437 is in theshape of a spiral. Transparent electrodes may be made of indium tinoxide (ITO).

Additional electrodes may be used for quick reversion of the liquidcrystal to a non-shaped or natural state. A small control voltage isthus used to dynamically change the refractive index of the material thelight passes through. The voltage generates a spatially non-uniformmagnetic field of a desired shape, allowing the liquid crystal tofunction as a lens.

In one embodiment, the camera includes the black silicon, short waveinfrared (SWIR) CMOS sensor described elsewhere in this patent. Inanother embodiment, the camera is a 5 megapixel (MP)optically-stabilized video sensor. In one embodiment, the controlsinclude a 3 GHz microprocessor or microcontroller, and may also includea 633 MHz digital signal processor with a 30 M polygon/second graphicaccelerator for real-time image processing for images from the camera orvideo sensor. In one embodiment, the augmented reality glasses mayinclude a wireless internet, radio or telecommunications capability forwideband, personal area network (PAN), local area network (LAN), a widelocal area network, WLAN, conforming to IEEE 802.11, or reach-backcommunications. The equipment furnished in one embodiment includes aBluetooth capability, conforming to IEEE 802.15. In one embodiment, theaugmented reality glasses include an encryption system, such as a256-bit Advanced Encryption System (AES) encryption system or othersuitable encryption program, for secure communications.

In one embodiment, the wireless telecommunications may include acapability for a 3G or 4G network and may also include a wirelessinternet capability. In order for an extended life, the augmentedreality eyepiece or glasses may also include at least one lithium-ionbattery, and as discussed above, a recharging capability. The rechargingplug may comprise an AC/DC power converter and may be capable of usingmultiple input voltages, such as 120 or 240 VAC. The controls foradjusting the focus of the adjustable focus lenses in one embodimentcomprises a 2D or 3D wireless air mouse or other non-contact controlresponsive to gestures or movements of the user. A 2D mouse is availablefrom Logitech, Fremont, Calif., USA. A 3D mouse is described herein, orothers such as the Cideko AVK05 available from Cideko, Taiwan, R.O.C,may be used.

In an embodiment, the eyepiece may comprise electronics suitable forcontrolling the optics, and associated systems, including a centralprocessing unit, non-volatile memory, digital signal processors, 3-Dgraphics accelerators, and the like. The eyepiece may provide additionalelectronic elements or features, including inertial navigation systems,cameras, microphones, audio output, power, communication systems,sensors, stopwatch or chronometer functions, thermometer, vibratorytemple motors, motion sensor, a microphone to enable audio control ofthe system, a UV sensor to enable contrast and dimming with photochromicmaterials, and the like.

In an embodiment, the central processing unit (CPU) of the eyepiece maybe an OMAP 4, with dual 1 GHz processor cores. The CPU may include a 633MHz DSP, giving a capability for the CPU of 30 million polygons/second.

The system may also provide dual micro-SD (secure digital) slots forprovisioning of additional removable non-volatile memory.

An on-board camera may provide 1.3 MP color and record up to 60 minutesof video footage. The recorded video may be transferred wirelessly orusing a mini-USB transfer device to off-load footage.

The communications system-on-a-chip (SOC) may be capable of operatingwith wide local area networks (WLAN), Bluetooth version 3.0, a GPSreceiver, an FM radio, and the like.

The eyepiece may operate on a 3.6 VDC lithium-ion rechargeable batteryfor long battery life and ease of use. An additional power source may beprovided through solar cells on the exterior of the frame of the system.These solar cells may supply power and may also be capable of rechargingthe lithium-ion battery.

The total power consumption of the eyepiece may be approximately 400 mW,but is variable depending on features and applications used. Forexample, processor-intensive applications with significant videographics demand more power, and will be closer to 400 mW. Simpler, lessvideo-intensive applications will use less power. The operation time ona charge also may vary with application and feature usage.

The micro-projector illumination engine, also known herein as theprojector, may include multiple light emitting diodes (LEDs). In orderto provide life-like color, Osram red, Cree green, and Cree blue LEDsare used. These are die-based LEDs. The RGB engine may provide anadjustable color output, allowing a user to optimize viewing for variousprograms and applications.

In embodiments, illumination may be added to the glasses or controlledthrough various means. For example, LED lights or other lights may beembedded in the frame of the eyepiece, such as in the nose bridge,around the composite lens, or at the temples.

The intensity of the illumination and or the color of illumination maybe modulated. Modulation may be accomplished through the various controltechnologies described herein, through various applications, filteringand magnification.

By way of example, illumination may be modulated through various controltechnologies described herein such as through the adjustment of acontrol knob, a gesture, eye movement, or voice command. If a userdesires to increase the intensity of illumination, the user may adjust acontrol knob on the glasses or he may adjust a control knob in the userinterface displayed on the lens or by other means. The user may use eyemovements to control the knob displayed on the lens or he may controlthe knob by other means. The user may adjust illumination through amovement of the hand or other body movement such that the intensity orcolor of illumination changes based on the movement made by the user.Also, the user may adjust the illumination through a voice command suchas by speaking a phrase requesting increased or decreased illuminationor requesting other colors to be displayed. Additionally, illuminationmodulation may be achieved through any control technology describedherein or by other means.

Further, the illumination may be modulated per the particularapplication being executed. As an example, an application mayautomatically adjust the intensity of illumination or color ofillumination based on the optimal settings for that application. If thecurrent levels of illumination are not at the optimal levels for theapplication being executed, a message or command may be sent to providefor illumination adjustment.

In embodiments, illumination modulation may be accomplished throughfiltering and or through magnification. For example, filteringtechniques may be employed that allow the intensity and or color of thelight to be changed such that the optimal or desired illumination isachieved. Also, in embodiments, the intensity of the illumination may bemodulated by applying greater or less magnification to reach the desiredillumination intensity.

The projector may be connected to the display to output the video andother display elements to the user. The display used may be an SVGA800×600 dots/inch SYNDIANT liquid crystal on silicon (LCoS) display.

The target MPE dimensions for the system may be 24 mm×12 mm×6 mm

The focus may be adjustable, allowing a user to refine the projectoroutput to suit their needs.

The optics system may be contained within a housing fabricated for6061-T6 aluminum and glass-filled ABS/PC.

The weight of the system, in an embodiment, is estimated to be 3.75ounces, or 95 grams.

In an embodiment, the eyepiece and associated electronics provide nightvision capability. This night vision capability may be enabled by ablack silicon SWIR sensor. Black silicon is a complementary metal-oxidesilicon (CMOS) processing technique that enhances the photo response ofsilicon over 100 times. The spectral range is expanded deep into theshort wave infra-red (SWIR) wavelength range. In this technique, a 300nm deep absorbing and anti-reflective layer is added to the glasses.This layer offers improved responsivity as shown in FIG. 11, where theresponsivity of black silicon is much greater than silicon's over thevisible and NIR ranges and extends well into the SWIR range. Thistechnology is an improvement over current technology, which suffers fromextremely high cost, performance issues, as well as high volumemanufacturability problems. Incorporating this technology into nightvision optics brings the economic advantages of CMOS technology into thedesign.

Unlike current night-vision goggles (NVGs), which amplify starlight orother ambient light from the visible light spectrum, SWIR sensors pickup individual photons and convert light in the SWIR spectrum toelectrical signals, similar to digital photography. The photons can beproduced from the natural recombination of oxygen and hydrogen atoms inthe atmosphere at night, also referred to as “Night Glow.” Shortwaveinfrared devices see objects at night by detecting the invisible,shortwave infrared radiation within reflected star light, city lights orthe moon. They also work in daylight, or through fog, haze or smoke,whereas the current NVG Image Intensifier infrared sensors would beoverwhelmed by heat or brightness. Because shortwave infrared devicespick up invisible radiation on the edge of the visible spectrum, theSWIR images look like the images produced by visible light with the sameshadows and contrast and facial details, only in black and white,dramatically enhancing recognition so people look like people; theydon't look like blobs often seen with thermal Imagers. One of theimportant SWIR capabilities is of providing views of targeting lasers onthe battlefield. Targeting lasers (1.064 um) are not visible withcurrent night-vision goggles. With SWIR Electro-optics, soldiers will beable to view every targeting laser in use, including those used by theenemy. Unlike Thermal Imagers, which do not penetrate windows onvehicles or buildings, the Visible/Near Infrared/Short Wave InfraredSensor can see through them—day or night, giving users an importanttactical advantage.

Certain advantages include using active illumination only when needed.In some instances there may be sufficient natural illumination at night,such as during a full moon. When such is the case, artificial nightvision using active illumination may not be necessary. With blacksilicon CMOS-based SWIR sensors, active illumination may not be neededduring these conditions, and is not provided, thus improving batterylife.

In addition, a black silicon image sensor may have over eight times thesignal to noise ratio found in costly indium-gallium arsenide imagesensors under night sky conditions. Better resolution is also providedby this technology, offering much higher resolution than available usingcurrent technology for night vision. Typically, long wavelength imagesproduced by CMOS-based SWIR have been difficult to interpret, havinggood heat detection, but poor resolution. This problem is solved with ablack image silicon SWIR sensor, which relies on much shorterwavelengths. SWIR is highly desirable for battlefield night visionglasses for these reasons. FIG. 12 illustrates the effectiveness ofblack silicon night vision technology, providing both before and afterimages of seeing through a) dust; b) fog, and c) smoke. The images inFIG. 12 demonstrate the performance of the new VIS/NIR/SWIR blacksilicon sensor. In embodiments, the image sensor may be able todistinguish between changes in the natural environment, such asdisturbed vegetation, disturbed ground, and the like. For example, anenemy combatant may have recently placed an explosive device in theground, and so the ground over the explosive will be ‘disturbed ground’,and the image sensor (along with processing facilities internal orexternal to the eyepiece) may be able to distinguish the recentlydisturbed ground from the surrounding ground. In this way, a soldier maybe able to detect the possible placement of an underground explosivedevice (e.g. an improvised explosive device (IED)) from a distance.

Previous night vision systems suffered from “blooms” from bright lightsources, such as streetlights. These “blooms” were particularly strongin image intensifying technology and are also associated with a loss ofresolution. In some cases, cooling systems are necessary in imageintensifying technology systems, increasing weight and shorteningbattery power lifespan. FIG. 17 shows the difference in image qualitybetween A) a flexible platform of uncooled CMOS image sensors capable ofVIS/NIR/SWIR imaging and B) an image intensified night vision system.

FIG. 13 depicts the difference in structure between current or incumbentvision enhancement technology 1300 and uncooled CMOS image sensors 1307.The incumbent platform (FIG. 13A) limits deployment because of cost,weight, power consumption, spectral range, and reliability issues.Incumbent systems are typically comprised of a front lens 1301,photocathode 1302, micro channel plate 1303, high voltage power supply1304, phosphorous screen 1305, and eyepiece 1306. This is in contrast toa flexible platform (FIG. 13B) of uncooled CMOS image sensors 1307capable of VIS/NIR/SWIR imaging at a fraction of the cost, powerconsumption, and weight. These much simpler sensors include a front lens1308 and an image sensor 1309 with a digital image output.

These advantages derive from the CMOS compatible processing techniquethat enhances the photo response of silicon over 100 times and extendsthe spectral range deep into the short wave infrared region. Thedifference in responsivity is illustrated in FIG. 13C. While typicalnight vision goggles are limited to the UV, visible and near infrared(NIR) ranges, to about 1100 nm (1.1 micrometers) the newer CMOS imagesensor ranges also include the short wave infrared (SWIR) spectrum, outto as much as 2000 nm (2 micrometers).

The black silicon core technology may offer significant improvement overcurrent night vision glasses. Femtosecond laser doping may enhance thelight detection properties of silicon across a broad spectrum.Additionally, optical response may be improved by a factor of 100 to10,000. The black silicon technology is a fast, scalable, and CMOScompatible technology at a very low cost, compared to current nightvision systems. Black silicon technology may also provide a lowoperation bias, with 3.3 V typical. In addition, uncooled performancemay be possible up to 50° C. Cooling requirements of current technologyincrease both weight and power consumption, and also create discomfortin users. As noted above, the black silicon core technology offers ahigh-resolution replacement for current image intensifier technology.Black silicon core technology may provide high speed electronicshuttering at speeds up to 1000 frames/second with minimal cross talk.In certain embodiments of the night vision eyepiece, an OLED display maybe preferred over other optical displays, such as the LCoS display.

The eyepiece incorporating the VIS/NIR/SWIR black silicon sensor mayprovide for better situational awareness (SAAS) surveillance andreal-time image enhancement.

In some embodiments, the VIS/NIR/SWIR black silicon sensor may beincorporated into a form factor suitable for night vision only, such asa night vision goggle or a night vision helmet. The night vision gogglemay include features that make it suitable for the military market, suchas ruggedization and alternative power supplies, while other formfactors may be suitable for the consumer or toy market. In one example,the night vision goggles may have extended range, such as 500-1200 nm,and may also useable as a camera.

In some embodiments, the VIS/NIR/SWIR black silicon sensor as well asother outboard sensors may be incorporated into a mounted camera thatmay be mounted on transport or combat vehicles so that the real-timefeed can be sent to the driver or other occupants of the vehicle bysuperimposing the video on the forward view without obstructing it. Thedriver can better see where he or she is going, the gunner can bettersee threats or targets of opportunity, and the navigator can bettersense situational awareness (SAAS) while also looking for threats. Thefeed could also be sent to off-site locations as desired, such as higherheadquarters of memory/storage locations for later use in targeting,navigation, surveillance, data mining, and the like.

Further advantages of the eyepiece may include robust connectivity. Thisconnectivity enables download and transmission using Bluetooth,Wi-Fi/Internet, cellular, satellite, 3G, FM/AM, TV, and UVB transceiverfor sending/receiving vast amounts of data quickly. For example, the UWBtransceiver may be used to create a very high data rate,low-probability-of-intercept/low-probability-of-detection (LPI/LPD),Wireless Personal Area Network (WPAN) to connect weapons sights,weapons-mounted mouse/controller, E/O sensors, medical sensors,audio/video displays, and the like. In other embodiments, the WPAN maybe created using other communications protocols. For example, a WPANtransceiver may be a COTS-compliant module front end to make the powermanagement of a combat radio highly responsive and to avoid jeopardizingthe robustness of the radio. By integrating the ultra wideband (UWB)transceiver, baseband/MAC and encryption chips onto a module, aphysically small dynamic and configurable transceiver to addressmultiple operational needs is obtained. The WPAN transceivers create alow power, encrypted, wireless personal area network (WPAN) betweensoldier worn devices. The WPAN transceivers can be attached or embeddedinto nearly any fielded military device with a network interface(handheld computers, combat displays, etc). The system is capable ofsupporting many users, AES encryption, robust against jamming and RFinterference as well as being ideal for combat providing lowprobabilities of interception and detection (LPI/LPD). The WPANtransceivers eliminate the bulk, weight and “snagability” of data cableson the soldier. Interfaces include USB 1.1, USB 2.0 OTG, Ethernet 10-,100 Base-T and RS232 9-pin D-Sub. The power output may be −10, −20 dBmoutputs for a variable range of up to 2 meters. The data capacity may be768 Mbps and greater. The bandwidth may be 1.7 GHz. Encryption may be128-bit, 192-bit or 256-bit AES. The WPAN transceiver may includeOptimized Message Authentication Code (MAC) generation. The WPANtransceiver may comply to MIL-STD-461F. The WPAN transceiver may be inthe form of a connector dust cap and may attach to any fielded militarydevice. The WPAN transceiver allows simultaneous video, voice, stills,text and chat, eliminates the need for data cables between electronicdevices, allows hands-free control of multiple devices withoutdistraction, features an adjustable connectivity range, interfaces withEthernet and USB 2.0, features an adjustable frequency 3.1 to 10.6 GHzand 200 mw peak draw and nominal standby.

For example, the WPAN transceiver may enable creating a WPAN between theeyepiece 100 in the form of a GSE stereo heads-up combat displayglasses, a computer, a remote computer controller, and biometricenrollment devices like that seen in FIG. 58. In another example, theWPAN transceiver may enable creating a WPAN between the eyepiece in theform of flip-up/-down heads-up display combat glasses, the HUD CPU (ifit is external), a weapon fore-grip controller, and a forearm computersimilar to that seen in FIG. 58.

The eyepiece may provide its own cellular connectivity, such as though apersonal wireless connection with a cellular system. The personalwireless connection may be available for only the wearer of theeyepiece, or it may be available to a plurality of proximate users, suchas in a Wi-Fi hot spot (e.g. MiFi), where the eyepiece provides a localhotspot for others to utilize. These proximate users may be otherwearers of an eyepiece, or users of some other wireless computingdevice, such as a mobile communications facility (e.g. mobile phone).Through this personal wireless connection, the wearer may not need othercellular or Internet wireless connections to connect to wirelessservices. For instance, without a personal wireless connectionintegrated into the eyepiece, the wearer may have to find a WiFiconnection point or tether to their mobile communications facility inorder to establish a wireless connection. In embodiments, the eyepiecemay be able to replace the need for having a separate mobilecommunications device, such as a mobile phone, mobile computer, and thelike, by integrating these functions and user interfaces into theeyepiece. For instance, the eyepiece may have an integrated WiFiconnection or hotspot, a real or virtual keyboard interface, a USB hub,speakers (e.g. to stream music to) or speaker input connections,integrated camera, external camera, and the like. In embodiments, anexternal device, in connectivity with the eyepiece, may provide a singleunit with a personal network connection (e.g. WiFi, cellularconnection), keyboard, control pad (e.g. a touch pad), and the like.

Communications from the eyepiece may include communication links forspecial purposes. For instance, an ultra-wide bandwidth communicationslink may be utilized when sending and/or receiving large volumes of datain a short amount of time. In another instance, a near-fieldcommunications (NFC) link may be used with very limited transmissionrange in order to post information to transmit to personnel when theyare very near, such as for tactical reasons, for local directions, forwarnings, and the like. For example, a soldier may be able to post/holdinformation securely, and transmit only to people very near by with aneed-to-know or need-to-use the information. In another instance, awireless personal area network (PAN) may be utilized, such as to connectweapons sights, weapons-mounted mouse/controller, electro-optic sensors,medical sensors, audio-visual displays, and the like.

The eyepiece may include MEMS-based inertial navigation systems, such asa GPS processor, an accelerometer (e.g. for enabling head control of thesystem and other functions), a gyroscope, an altimeter, an inclinometer,a speedometer/odometer, a laser rangefinder, and a magnetometer, whichalso enables image stabilization.

The eyepiece may include integrated headphones, such as the articulatingearbud 120, that provide audio output to the user or wearer.

In an embodiment, a forward facing camera (see FIG. 21) integrated withthe eyepiece may enable basic augmented reality. In augmented reality, aviewer can image what is being viewed and then layer an augmented,edited, tagged, or analyzed version on top of the basic view. In thealternative, associated data may be displayed with or over the basicimage. If two cameras are provided and are mounted at the correctinterpupillary distance for the user, stereo video imagery may becreated. This capability may be useful for persons requiring visionassistance. Many people suffer from deficiencies in their vision, suchas near-sightedness, far-sightedness, and so forth. A camera and a veryclose, virtual screen as described herein provides a “video” for suchpersons, the video adjustable in terms of focal point, nearer orfarther, and fully in control by the person via voice or other command.This capability may also be useful for persons suffering diseases of theeye, such as cataracts, retinitis pigmentosa, and the like. So long assome organic vision capability remains, an augmented reality eyepiececan help a person see more clearly. Embodiments of the eyepiece mayfeature one or more of magnification, increased brightness, and abilityto map content to the areas of the eye that are still healthy.Embodiments of the eyepiece may be used as bifocals or a magnifyingglass. The wearer may be able to increase zoom in the field of view orincrease zoom within a partial field of view. In an embodiment, anassociated camera may make an image of the object and then present theuser with a zoomed picture. A user interface may allow a wearer to pointat the area that he wants zoomed, such as with the control techniquesdescribed herein, so the image processing can stay on task as opposed tojust zooming in on everything in the camera's field of view.

A rear-facing camera (not shown) may also be incorporated into theeyepiece in a further embodiment. In this embodiment, the rear-facingcamera may enable eye control of the eyepiece, with the user makingapplication or feature selection by directing his or her eyes to aspecific item displayed on the eyepiece.

A further embodiment of a device for capturing biometric data aboutindividuals may incorporate a microcassegrain telescoping folded opticcamera into the device. The microcassegrain telescoping folded opticcamera may be mounted on a handheld device, such as the bio-printdevice, the bio-phone, and could also be mounted on glasses used as partof a bio-kit to collect biometric data.

A cassegrain reflector is a combination of a primary concave mirror anda secondary convex mirror. These reflectors are often used in opticaltelescopes and radio antennas because they deliver good light (or sound)collecting capability in a shorter, smaller package.

In a symmetrical cassegrain both mirrors are aligned about the opticalaxis, and the primary mirror usually has a hole in the center, allowinglight to reach the eyepiece or a camera chip or light detection device,such as a CCD chip. An alternate design, often used in radio telescopes,places the final focus in front of the primary reflector. A furtheralternate design may tilt the mirrors to avoid obstructing the primaryor secondary mirror and may eliminate the need for a hole in the primarymirror or secondary mirror. The microcassegrain telescoping folded opticcamera may use any of the above variations, with the final selectiondetermined by the desired size of the optic device.

The classic cassegrain configuration 3500 uses a parabolic reflector asthe primary mirror and a hyperbolic mirror as the secondary mirror.Further embodiments of the microcassegrain telescoping folded opticcamera may use a hyperbolic primary mirror and/or a spherical orelliptical secondary mirror. In operation the classic cassegrain with aparabolic primary mirror and a hyperbolic secondary mirror reflects thelight back down through a hole in the primary, as shown in FIG. 35.Folding the optical path makes the design more compact, and in a “micro”size, suitable for use with the bio-print sensor and bio-print kitdescribed herein. In a folded optic system, the beam is bent to make theoptical path much longer than the physical length of the system. Onecommon example of folded optics is prismatic binoculars. In a cameralens the secondary mirror may be mounted on an optically flat, opticallyclear glass plate that closes the lens tube. This support eliminates“star-shaped” diffraction effects that are caused by a straight-vanedsupport spider. This allows for a sealed closed tube and protects theprimary mirror, albeit at some loss of light collecting power.

The cassegrain design also makes use of the special properties ofparabolic and hyperbolic reflectors. A concave parabolic reflector willreflect all incoming light rays parallel to its axis of symmetry to asingle focus point. A convex hyperbolic reflector has two foci andreflects all light rays directed at one focus point toward the otherfocus point. Mirrors in this type of lens are designed and positioned toshare one focus, placing the second focus of the hyperbolic mirror atthe same point as where the image is observed, usually just outside theeyepiece. The parabolic mirror reflects parallel light rays entering thelens to its focus, which is coincident with the focus of the hyperbolicmirror. The hyperbolic mirror then reflects those light rays to theother focus point, where the camera records the image.

FIG. 36 shows the configuration of the microcassegrain telescopingfolded optic camera. The camera may be mounted on augmented realityglasses, a bio-phone, or other biometric collection device. Theassembly, 3600 has multiple telescoping segments that allow the camerato extend with cassegrain optics providing for a longer optical path.Threads 3602 allow the camera to be mounted on a device, such asaugmented reality glasses or other biometric collection device. Whilethe embodiment depicted in FIG. 36 uses threads, other mounting schemessuch as bayonet mount, knobs, or press-fit, may also be used. A firsttelescoping section 3604 also acts as an external housing when the lensis in the fully retracted position. The camera may also incorporate amotor to drive the extension and retraction of the camera. A secondtelescoping section 3606 may also be included. Other embodiments mayincorporate varying numbers of telescoping sections, depending on thelength of optical path needed for the selected task or data to becollected. A third telescoping section 3608 includes the lens and areflecting mirror. The reflecting mirror may be a primary reflector ifthe camera is designed following classic cassegrain design. Thesecondary mirror may be contained in first telescoping section 3604.

Further embodiments may utilize microscopic mirrors to form the camera,while still providing for a longer optical path through the use offolded optics. The same principles of cassegrain design are used.

Lens 3610 provides optics for use in conjunction with the folded opticsof the cassegrain design. The lens 3610 may be selected from a varietyof types, and may vary depending on the application. The threads 3602permit a variety of cameras to be interchanged depending on the needs ofthe user.

Eye control of feature and option selection may be controlled andactivated by object recognition software loaded on the system processor.Object recognition software may enable augmented reality, combine therecognition output with querying a database, combine the recognitionoutput with a computational tool to determine dependencies/likelihoods,and the like.

Three-dimensional viewing is also possible in an additional embodimentthat incorporates a 3D projector. Two stacked picoprojectors (not shown)may be used to create the three dimensional image output.

Referring to FIG. 10, a plurality of digital CMOS Sensors with redundantmicros and DSPs for each sensor array and projector detect visible, nearinfrared, and short wave infrared light to enable passive day and nightoperations, such as real-time image enhancement 1002, real-time keystonecorrection 1004, and real-time virtual perspective correction 1008. Theeyepiece may utilize digital CMOS image sensors and directionalmicrophones (e.g. microphone arrays) as described herein, such as forvisible imaging for monitoring the visible scene (e.g. for biometricrecognition, gesture control, coordinated imaging with 2D/3D projectedmaps), IR/UV imaging for scene enhancement (e.g. seeing through haze,smoke, in the dark), sound direction sensing (e.g. the direction of agunshot or explosion, voice detection), and the like. In embodiments,each of these sensor inputs may be fed to a digital signal processor(DSP) for processing, such as internal to the eyepiece or as interfacedto external processing facilities. The outputs of the DSP processing ofeach sensor input stream may then be algorithmically combined in amanner to generate useful intelligence data. For instance, this systemmay be useful for a combination of real-time facial recognition, realtime voice detection, and analysis through links to a database,especially with distortion corrections and contemporaneous GPS locationfor soldiers, service personnel, and the like, such as in monitoringremote areas of interest, e.g., known paths or trails, or high-securityareas.

The augmented reality eyepiece or glasses may be powered by any storedenergy system, such as battery power, solar power, line power, and thelike. A solar energy collector may be placed on the frame, on a beltclip, and the like. Battery charging may occur using a wall charger, carcharger, on a belt clip, in a glasses case, and the like. In oneembodiment, the eyepiece may be rechargeable and be equipped with amini-USB connector for recharging. In another embodiment, the eyepiecemay be equipped for remote inductive recharging by one or more remoteinductive power conversion technologies, such as those provided byPowercast, Ligonier, Pa., USA; and Fulton Int'l Inc., Ada, Mich., USA,which also owns another provider, Splashpower, Inc., Cambridge, UK.

The augmented reality eyepiece also includes a camera and any interfacenecessary to connect the camera to the circuit. The output of the cameramay be stored in memory and may also be displayed on the displayavailable to the wearer of the glasses. A display driver may also beused to control the display. The augmented reality device also includesa power supply, such as a battery, as shown, power management circuitsand a circuit for recharging the power supply. As noted elsewhere,recharging may take place via a hard connection, e.g., a mini-USBconnector, or by means of an inductor, a solar panel input, and soforth.

The control system for the eyepiece or glasses may include a controlalgorithm for conserving power when the power source, such as a battery,indicates low power. This conservation algorithm may include shuttingpower down to applications that are energy intensive, such as lighting,a camera, or sensors that require high levels of energy, such as anysensor requiring a heater, for example. Other conservation steps mayinclude slowing down the power used for a sensor or for a camera, e.g.,slowing the sampling or frame rates, going to a slower sampling or framerate when the power is low; or shutting down the sensor or camera at aneven lower level. Thus, there may be at least three operating modesdepending on the available power: a normal mode; a conserve power mode;and an emergency or shutdown mode.

Applications of the present disclosure may be controlled throughmovements and direct actions of the wearer, such as movement of his orher hand, finger, feet, head, eyes, and the like, enabled throughfacilities of the eyepiece (e.g. accelerometers, gyros, cameras, opticalsensors, GPS sensors, and the like) and/or through facilities worn ormounted on the wearer (e.g. body mounted sensor control facilities). Inthis way, the wearer may directly control the eyepiece through movementsand/or actions of their body without the use of a traditional hand-heldremote controller. For instance, the wearer may have a sense device,such as a position sense device, mounted on one or both hands, such ason at least one finger, on the palm, on the back of the hand, and thelike, where the position sense device provides position data of thehand, and provides wireless communications of position data as commandinformation to the eyepiece. In embodiments, the sense device of thepresent disclosure may include a gyroscopic device (e.g. electronicgyroscope, MEMS gyroscope, mechanical gyroscope, quantum gyroscope, ringlaser gyroscope, fiber optic gyroscope), accelerometers, MEMSaccelerometers, velocity sensors, force sensors, pressure sensors,optical sensors, proximity sensor, RFID, and the like, in the providingof position information. For example, a wearer may have a position sensedevice mounted on their right index finger, where the device is able tosense motion of the finger. In this example, the user may activate theeyepiece either through some switching mechanism on the eyepiece orthrough some predetermined motion sequence of the finger, such as movingthe finger quickly, tapping the finger against a hard surface, and thelike. Note that tapping against a hard surface may be interpretedthrough sensing by accelerometers, force sensors, pressure sensors, andthe like. The position sense device may then transmit motions of thefinger as command information, such as moving the finger in the air tomove a cursor across the displayed or projected image, moving in quickmotion to indicate a selection, and the like. In embodiments, theposition sense device may send sensed command information directly tothe eyepiece for command processing, or the command processing circuitrymay be co-located with the position sense device, such as in thisexample, mounted on the finger as part of an assembly including thesensors of the position sense device.

In embodiments, the wearer may have a plurality of position sensedevices mounted on their body. For instance, and in continuation of thepreceding example, the wearer may have position sense devices mounted ona plurality of points on the hand, such as with individual sensors ondifferent fingers, or as a collection of devices, such as in a glove. Inthis way, the aggregate sense command information from the collection ofsensors at different locations on the hand may be used to provide morecomplex command information. For instance, the wearer may use a sensordevice glove to play a game, where the glove senses the grasp and motionof the user's hands on a ball, bat, racket, and the like, in the use ofthe present disclosure in the simulation and play of a simulated game.In embodiments, the plurality of position sense devices may be mountedon different parts of the body, allowing the wearer to transmit complexmotions of the body to the eyepiece for use by an application.

In embodiments, the sense device may have a force sensor, pressuresensor, and the like, such as for detecting when the sense device comesin contact with an object. For instance, a sense device may include aforce sensor at the tip of a wearer's finger. In this case, the wearermay tap, multiple tap, sequence taps, swipe, touch, and the like togenerate a command to the eyepiece. Force sensors may also be used toindicate degrees of touch, grip, push, and the like, where predeterminedor learned thresholds determine different command information. In thisway, commands may be delivered as a series of continuous commands thatconstantly update the command information being used in an applicationthrough the eyepiece. In an example, a wearer may be running asimulation, such as a game application, military application, commercialapplication, and the like, where the movements and contact with objects,such as through at least one of a plurality of sense devices, are fed tothe eyepiece as commands that influence the simulation displayed throughthe eyepiece. For instance, a sense device may be included in a pencontroller, where the pen controller may have a force sensor, pressuresensor, inertial measurement unit, and the like, and where the pencontroller may be used to produce virtual writing, control a cursorassociated with the eyepiece's display, act as a computer mouse, providecontrol commands though physical motion and/or contact, and the like.

In embodiments, the sense device may include an optical sensor oroptical transmitter as a way for movement to be interpreted as acommand. For instance, a sense device may include an optical sensormounted on the hand of the wearer, and the eyepiece housing may includean optical transmitter, such that when a user moves their hand past theoptical transmitter on the eyepiece, the motions may be interpreted ascommands. A motion detected through an optical sensor may includeswiping past at different speeds, with repeated motions, combinations ofdwelling and movement, and the like. In embodiments, optical sensorsand/or transmitters may be located on the eyepiece, mounted on thewearer (e.g. on the hand, foot, in a glove, piece of clothing), or usedin combinations between different areas on the wearer and the eyepiece,and the like.

In one embodiment, a number of sensors useful for monitoring thecondition of the wearer or a person in proximity to the wearer aremounted within the augmented reality glasses. Sensors have become muchsmaller, thanks to advances in electronics technology. Signaltransducing and signal processing technologies have also made greatprogress in the direction of size reduction and digitization.Accordingly, it is possible to have not merely a temperature sensor inthe AR glasses, but an entire sensor array. These sensors may include,as noted, a temperature sensor, and also sensor to detect: pulse rate;beat-to-beat heart variability; EKG or ECG; respiration rate; core bodytemperature; heat flow from the body; galvanic skin response or GSR;EMG; EEG; EOG; blood pressure; body fat; hydration level; activitylevel; oxygen consumption; glucose or blood sugar level; body position;and UV radiation exposure or absorption. In addition, there may also bea retinal sensor and a blood oxygenation sensor (such as an Sp0₂sensor), among others. Such sensors are available from a variety ofmanufacturers, including Vermed, Bellows Falls, Vt., USA; VTI, Ventaa,Finland; and ServoFlow, Lexington, Mass., USA.

In some embodiments, it may be more useful to have sensors mounted onthe person or on equipment of the person, rather than on the glassesthemselves. For example, accelerometers, motion sensors and vibrationsensors may be usefully mounted on the person, on clothing of theperson, or on equipment worn by the person. These sensors may maintaincontinuous or periodic contact with the controller of the AR glassesthrough a Bluetooth® radio transmitter or other radio device adhering toIEEE 802.11 specifications. For example, if a physician wishes tomonitor motion or shock experienced by a patient during a foot race, thesensors may be more useful if they are mounted directly on the person'sskin, or even on a T-shirt worn by the person, rather than mounted onthe glasses. In these cases, a more accurate reading may be obtained bya sensor placed on the person or on the clothing rather than on theglasses. Such sensors need not be as tiny as the sensors which would besuitable for mounting on the glasses themselves, and be more useful, asseen.

The AR glasses or goggles may also include environmental sensors orsensor arrays. These sensors are mounted on the glasses and sample theatmosphere or air in the vicinity of the wearer. These sensors or sensorarray may be sensitive to certain substances or concentrations ofsubstances. For example, sensors and arrays are available to measureconcentrations of carbon monoxide, oxides of nitrogen (“NO_(x)”),temperature, relative humidity, noise level, volatile organic chemicals(VOC), ozone, particulates, hydrogen sulfide, barometric pressure andultraviolet light and its intensity. Vendors and manufacturers include:Sensares, Crolles, FR; Cairpol, Ales, FR; Critical EnvironmentalTechnologies of Canada, Delta, B.C., Canada; Apollo Electronics Co.,Shenzhen, China; and AV Technology Ltd., Stockport, Cheshire, UK. Manyother sensors are well known. If such sensors are mounted on the personor on clothing or equipment of the person, they may also be useful.These environmental sensors may include radiation sensors, chemicalsensors, poisonous gas sensors, and the like.

In one embodiment, environmental sensors, health monitoring sensors, orboth, are mounted on the frames of the augmented reality glasses. Inanother embodiment, the sensors may be mounted on the person or onclothing or equipment of the person. For example, a sensor for measuringelectrical activity of a heart of the wearer may be implanted, withsuitable accessories for transducing and transmitting a signalindicative of the person's heart activity.

The signal may be transmitted a very short distance via a Bluetooth®radio transmitter or other radio device adhering to IEEE 802.15.1specifications. Other frequencies or protocols may be used instead. Thesignal may then be processed by the signal-monitoring and processingequipment of the augmented reality glasses, and recorded and displayedon the virtual screen available to the wearer. In another embodiment,the signal may also be sent via the AR glasses to a friend or squadleader of the wearer. Thus, the health and well-being of the person maybe monitored by the person and by others, and may also be tracked overtime.

In another embodiment, environmental sensors may be mounted on theperson or on equipment of the person. For example, radiation or chemicalsensors may be more useful if worn on outer clothing or a web-belt ofthe person, rather than mounted directly on the glasses. As noted above,signals from the sensors may be monitored locally by the person throughthe AR glasses. The sensor readings may also be transmitted elsewhere,either on demand or automatically, perhaps at set intervals, such asevery quarter-hour or half-hour. Thus, a history of sensor readings,whether of the person's body readings or of the environment, may be madefor tracking or trending purposes.

In an embodiment, an RF/micropower impulse radio (MIR) sensor may beassociated with the eyepiece and serve as a short-range medical radar.The sensor may operate on an ultra-wide band. The sensor may include anRF/impulse generator, receiver, and signal processor, and may be usefulfor detecting and measuring cardiac signals by measuring ion flow incardiac cells within 3 mm of the skin. The receiver may be a phasedarray antenna to enable determining a location of the signal in a regionof space. The sensor may be used to detect and identify cardiac signalsthrough blockages, such as walls, water, concrete, dirt, metal, wood,and the like. For example, a user may be able to use the sensor todetermine how many people are located in a concrete structure by howmany heart rates are detected. In another embodiment, a detected heartrate may serve as a unique identifier for a person so that they may berecognized in the future. In an embodiment, the RF/impulse generator maybe embedded in one device, such as the eyepiece or some other device,while the receiver is embedded in a different device, such as anothereyepiece or device. In this way, a virtual “tripwire” may be createdwhen a heart rate is detected between the transmitter and receiver. Inan embodiment, the sensor may be used as an in-field diagnostic orself-diagnosis tool. EKG's may be analyzed and stored for future use asa biometric identifier. A user may receive alerts of sensed heart ratesignals and how many heart rates are present as displayed content in theeyepiece.

FIG. 29 depicts an embodiment 2900 of an augmented reality eyepiece orglasses with a variety of sensors and communication equipment. One ormore than one environmental or health sensors are connected to a sensorinterface locally or remotely through a short range radio circuit and anantenna, as shown. The sensor interface circuit includes all devices fordetecting, amplifying, processing and sending on or transmitting thesignals detected by the sensor(s). The remote sensors may include, forexample, an implanted heart rate monitor or other body sensor (notshown). The other sensors may include an accelerometer, an inclinometer,a temperature sensor, a sensor suitable for detecting one or morechemicals or gasses, or any of the other health or environmental sensorsdiscussed in this disclosure. The sensor interface is connected to themicroprocessor or microcontroller of the augmented reality device, fromwhich point the information gathered may be recorded in memory, such asrandom access memory (RAM) or permanent memory, read only memory (ROM),as shown.

In an embodiment, a sense device enables simultaneous electric fieldsensing through the eyepiece. Electric field (EF) sensing is a method ofproximity sensing that allows computers to detect, evaluate and workwith objects in their vicinity. Physical contact with the skin, such asa handshake with another person or some other physical contact with aconductive or a non-conductive device or object, may be sensed as achange in an electric field and either enable data transfer to or fromthe eyepiece or terminate data transfer. For example, videos captured bythe eyepiece may be stored on the eyepiece until a wearer of theeyepiece with an embedded electric field sensing transceiver touches anobject and initiates data transfer from the eyepiece to a receiver. Thetransceiver may include a transmitter that includes a transmittercircuit that induces electric fields toward the body and a data sensecircuit, which distinguishes transmitting and receiving modes bydetecting both transmission and reception data and outputs controlsignals corresponding to the two modes to enable two-way communication.An instantaneous private network between two people may be generatedwith a contact, such as a handshake. Data may be transferred between aneyepiece of a user and a data receiver or eyepiece of the second user.Additional security measures may be used to enhance the private network,such as facial or audio recognition, detection of eye contact,fingerprint detection, biometric entry, and the like.

In embodiments, there may be an authentication facility associated withaccessing functionality of the eyepiece, such as access to displayed orprojected content, access to restricted projected content, enablingfunctionality of the eyepiece itself (e.g. as through a login to accessfunctionality of the eyepiece) either in whole or in part, and the like.Authentication may be provided through recognition of the wearer'svoice, iris, retina, fingerprint, and the like, or other biometricidentifier. For example, the eyepiece or an associated controller mayhave an IR, ultrasonic or capacitive tactile sensor for receivingcontrol input related to authentication or other eyepiece functions. Acapacitance sensor can detect a fingerprint and launch an application orotherwise control an eyepiece function. Each finger has a differentfingerprint so each finger can be used to control different eyepiecefunctions or quick launch different applications or provide variouslevels of authentication. Capacitance does not work with gloves but anultrasonic sensor does and can be used in the same way to providebiometric authentication or control. Ultrasonic sensors useful in theeyepiece or associated controller include Sonavation's SonicTouch™technology used in Sonavation's SonicSlide™ sensors, which works byacoustically measuring the ridges and valleys of the fingerprint toimage the fingerprint in 256 shades of gray in order to discern theslightest fingerprint detail. The key imaging component of theSonicSlide™ sensor is the ceramic Micro-Electro Mechanical System (MEMS)piezoelectric transducer array that is made from a ceramic compositematerial.

The authentication system may provide for a database of biometric inputsfor a plurality of users such that access control may be provided foruse of the eyepiece based on policies and associated access privilegesfor each of the users entered into the database. The eyepiece mayprovide for an authentication process. For instance, the authenticationfacility may sense when a user has taken the eyepiece off, and requirere-authentication when the user puts it back on. This better ensuresthat the eyepiece only provides access to those users that areauthorized, and for only those privileges that the wearer is authorizedfor. In an example, the authentication facility may be able to detectthe presence of a user's eye or head as the eyepiece is put on. In afirst level of access, the user may only be able to accesslow-sensitivity items until authentication is complete. Duringauthentication, the authentication facility may identify the user, andlook up their access privileges. Once these privileges have beendetermined, the authentication facility may then provide the appropriateaccess to the user. In the case of an unauthorized user being detected,the eyepiece may maintain access to low-sensitivity items, furtherrestrict access, deny access entirely, and the like.

In an embodiment, a receiver may be associated with an object to enablecontrol of that object via touch by a wearer of the eyepiece, whereintouch enables transmission or execution of a command signal in theobject. For example, a receiver may be associated with a car door lock.When a wearer of the eyepiece touches the car, the car door may unlock.In another example, a receiver may be embedded in a medicine bottle.When the wearer of the eyepiece touches the medicine bottle, an alarmsignal may be initiated. In another example, a receiver may beassociated with a wall along a sidewalk. As the wearer of the eyepiecepasses the wall or touches the wall, advertising may be launched eitherin the eyepiece or on a video panel of the wall.

In an embodiment, when a wearer of the eyepiece initiates a physicalcontact, a WiFi exchange of information with a receiver may provide anindication that the wearer is connected to an online activity such as agame or may provide verification of identity in an online environment.In the embodiment, a representation of the person could change color orundergo some other visual indication in response to the contact.

In embodiments, the eyepiece may include a tactile interface as in FIG.14, such as to enable haptic control of the eyepiece, such as with aswipe, tap, touch, press, click, roll of a rollerball, and the like. Forinstance, the tactile interface 1402 may be mounted on the frame of theeyepiece 1400, such as on an arm, both arms, the nosepiece, the top ofthe frame, the bottom of the frame, and the like. In embodiments, thetactile interface 1402 may include controls and functionality similar toa computer mouse, with left and right buttons, a 2D position control padsuch as described herein, and the like. For example, the tactileinterface may be mounted on the eyepiece near the user's temple and actas a ‘temple mouse’ controller for the eyepiece projected content to theuser and may include a temple-mounted rotary selector and enter button.In another example, the tactile interface may be one or more vibratorytemple motors which may vibrate to alert or notify the user, such as todanger left, danger right, a medical condition, and the like. Thetactile interface may be mounted on a controller separate from theeyepiece, such as a worn controller hand-carried controller, and thelike. If there is an accelerometer in the controller then it may sensethe user tapping, such as on a keyboard, on their hand (either on thehand with the controller or tapping with the hand that has thecontroller), and the like. The wearer may then touch the tactileinterface in a plurality of ways to be interpreted by the eyepiece ascommands, such as by tapping one or multiple times on the interface, bybrushing a finger across the interface, by pressing and holding, bypressing more than one interface at a time, and the like. Inembodiments, the tactile interface may be attached to the wearer's body(e.g. their hand, arm, leg, torso, neck), their clothing, as anattachment to their clothing, as a ring 1500, as a bracelet, as anecklace, and the like. For example, the interface may be attached onthe body, such as on the back of the wrist, where touching differentparts of the interface provides different command information (e.g.touching the front portion, the back portion, the center, holding for aperiod of time, tapping, swiping, and the like). In embodiments, usercontact with the tactile interface may be interpreted through force,pressure, movement, and the like. For instance, the tactile interfacemay incorporate resistive touch technologies, capacitive touchtechnologies, proportional pressure touch technologies, and the like. Inan example, the tactile interface may utilize discrete resistive touchtechnologies where the application requires the interface to be simple,rugged, low power, and the like. In another example, the tactileinterface may utilize capacitive tough technologies where morefunctionality is required through the interface, such as thoughmovement, swiping, multi-point contacts, and the like. In anotherexample, the tactile interface may utilize pressure touch technologies,such as when variable pressure commanding is required. In embodiments,any of these, or like touch technologies, may be used in any tactileinterface as described herein.

In another example, the wearer may have an interface mounted in a ringas shown in FIG. 15, a hand piece, and the like, where the interface mayhave at least one of a plurality of command interface types, such as atactile interface, a position sensor device, and the like with wirelesscommand connection to the eyepiece. In an embodiment, the ring 1500 mayhave controls that mirror a computer mouse, such as buttons 1504 (e.g.functioning as a one-button, multi-button, and like mouse functions), a2D position control 1502, scroll wheel, and the like. The buttons 1504and 2D position control 1502 may be as shown in FIG. 15, where thebuttons are on the side facing the thumb and the 2D position controlleris on the top. Alternately, the buttons and 2D position control may bein other configurations, such as all facing the thumb side, all on thetop surface, or any other combination. The 2D position control 1502 maybe a 2D button position controller (e.g. such as the TrackPoint pointingdevice embedded in some laptop keyboards to control the position of themouse), a pointing stick, joystick, an optical track pad, an opto touchwheel, a touch screen, touch pad, track pad, scrolling track pad,trackball, any other position or pointing controller, and the like. Inembodiments, control signals from the tactile interface (such as thering tactile interface 1500) may be provided with a wired or wirelessinterface to the eyepiece, where the user is able to conveniently supplycontrol inputs, such as with their hand, thumb, finger, and the like.For example, the user may be able to articulate the controls with theirthumb, where the ring is worn on the user's index finger. Inembodiments, a method or system may provide an interactive head-mountedeyepiece worn by a user, wherein the eyepiece includes an opticalassembly through which the user views a surrounding environment anddisplayed content, a processor for handling content for display to theuser, and an integrated projector facility for projecting the content tothe optical assembly, and a control device worn on the body of the user,such as a hand of the user, including at least one control componentactuated by the user, and providing a control command from the actuationof the at least one control component to the processor as a commandinstruction. The command instruction may be directed to the manipulationof content for display to the user. The control device may be worn on afirst digit of the hand of the user, and the at least one controlcomponent may be actuated by a second digit of a hand of the user. Thefirst digit may be the index finger, the second digit the thumb, and thefirst and second digit on the same hand of the user. The control devicemay have at least one control component mounted on the index finger sidefacing the thumb. The at least one control component may be a button.The at least one control component may be a 2D position controller. Thecontrol device may have at least one button actuated control componentmounted on the index finger side facing the thumb, and a 2D positioncontroller actuated control component mounted on the top facing side ofthe index finger. The control components may be mounted on at least twodigits of the user's hand. The control device may be worn as a glove onthe hand of the user. The control device may be worn on the wrist of theuser. The at least one control component may be worn on at least onedigit of the hand, and a transmission facility may be worn separately onthe hand. The transmission facility may be worn on the wrist. Thetransmission facility may be worn on the back of the hand. The controlcomponent may be at least one of a plurality of buttons. The at leastone button may provide a function substantially similar to aconventional computer mouse button. Two of the plurality of buttons mayfunction substantially similar to primary buttons of a conventionaltwo-button computer mouse. The control component may be a scrollingwheel. The control component may be a 2D position control component. The2D position control component may be a button position controller,pointing stick, joystick, optical track pad, opto-touch wheel, touchscreen, touch pad, track pad, scrolling track pad, trackball, capacitivetouch screen, and the like. The 2D position control component may becontrolled with the user's thumb. The control component may be atouch-screen capable of implementing touch controls includingbutton-like functions and 2D manipulation functions. The controlcomponent may be actuated when the user puts on the projected processorcontent pointing and control device.

In embodiments, the wearer may have an interface mounted in a ring1500AA that includes a camera 1502AA, such as shown in FIG. 15AA. Inembodiments, the ring controller 1502AA may have control interface typesas described herein, such as through buttons 1504, 2D position control1502, 3D position control (e.g. utilizing accelerometers, gyros), andthe like. The ring controller 1500AA may then be used to controlfunctions within the eyepiece, such as controlling the manipulation ofthe projected display content to the wearer. In embodiments, the controlinterfaces 1502, 1504 may provide control aspects to the embedded camera1502AA, such as on/off, zoom, pan, focus, recording a still imagepicture, recording a video, and the like. Alternately, the functions maybe controlled through other control aspects of the eyepiece, such asthrough voice control, other tactile control interfaces, eye gazedetection as described herein, and the like. The camera may also haveautomatic control functions enabled, such as auto-focus, timedfunctions, face detection and/or tracking, auto-zoom, and the like. Forexample, the ring controller 1500AA with integrated camera 1502AA may beused to view the wearer 1508AA during a videoconference enabled throughthe eyepiece, where the wearer 1508AA may hold the ring controller (e.g.as mounted on their finger) out in order to allow the camera 1502AA aview of their face for transmission to at least one other participant onthe videoconference. Alternately, the wearer may take the ringcontroller 1500AA off and place it down on a surface 1510AA (e.g. atable top) such that the camera 1502AA has a view of the wearer. Animage of the wearer 1512AA may then be displayed on the display area1518AA of the eyepiece and transmitted to others on the videoconference,such as along with the images 1514AA of other participants on thevideoconference call. In embodiments, the camera 1502AA may provide formanual or automatic FOV 1504AA adjustment. For instance, the wearer mayset the ring controller 1500AA down on a surface 1510AA for use in avideo conference call, and the FOV 1504AA may be controlled eithermanually (e.g. through button controls 1502, 1504, voice control, othertactile interface) or automatically (e.g. though face recognition) inorder for the camera's FOV 1504AA to be directed to the wearer's face.The FOV 1504AA may be enabled to change as the wearer moves, such as bytracking by face recognition. The FOV 1504AA may also zoomed in/out toadjust to changes in the position of the wearer's face. In embodiments,the camera 1502AA may be used for a plurality of still and/or videoapplications, where the view of the camera is provided to the wearer onthe display area 1518AA of the eyepiece, and where storage may beavailable in the eyepiece for storing the images/videos, which may betransferred, communicated, and the like, from the eyepiece to someexternal storage facility, user, web-application, and the like. Inembodiments, a camera may be incorporated in a plurality of differentmobile devices, such as worn on the arm, hand, wrist, finger, and thelike, such as the watch 3202 with embedded camera 3200 as shown in FIGS.32-33. As with the ring controller 1502AA, any of these mobile devicesmay include manual and/or automatic functions as described for the ringcontroller 1502AA. In embodiments, the ring controller 1502AA may haveadditional sensors, embedded functions, control features, and the like,such as a fingerprint scanner, tactile feedback, and LCD screen, anaccelerometer, Bluetooth, and the like. For instance, the ringcontroller may provide for synchronized monitoring between the eyepieceand other control components, such as described herein.

In embodiments, the eyepiece may provide a system and method forproviding an image of the wearer to videoconference participants throughthe use of an external mirror, where the wearer views themselves in themirror and an image of themselves is captured through an integratedcamera of the eyepiece. The captured image may be used directly, or theimage may be flipped to correct for the image reversal of the mirror. Inan example, the wearer may enter into a videoconference with a pluralityof other people, where the wearer may be able to view live video imagesof the others though the eyepiece. By utilizing an ordinary mirror, andan integrated camera in the eyepiece, the user may be able to viewthemselves in the mirror, have the image captured by the integratedcamera, and provide the other people with a image of themselves forpurposes of the videoconference. This image may also be available to thewearer as a projected image to the eyepiece, such as in addition to theimages of the other people involved in the videoconference.

In embodiments, a control component may provide a surface-sensingcomponent in the control device for detecting motion across a surfacemay also be provided. The surface sensing component may be disposed onthe palmar side of the user's hand. The surface may be at least one of ahard surface, a soft surface, surface of the user's skin, surface of theuser's clothing, and the like. Providing control commands may betransmitted wirelessly, through a wired connection, and the like. Thecontrol device may control a pointing function associated with thedisplayed processor content. The pointing function may be control of acursor position; selection of displayed content, selecting and movingdisplayed content; control of zoom, pan, field of view, size, positionof displayed content; and the like. The control device may control apointing function associated with the viewed surrounding environment.The pointing function may be placing a cursor on a viewed object in thesurrounding environment. The viewed object's location position may bedetermined by the processor in association with a camera integrated withthe eyepiece. The viewed object's identification may be determined bythe processor in association with a camera integrated with the eyepiece.The control device may control a function of the eyepiece. The functionmay be associated with the displayed content. The function may be a modecontrol of the eyepiece. The control device may be foldable for ease ofstorage when not worn by the user. In embodiments, the control devicemay be used with external devices, such as to control the externaldevice in association with the eyepiece. External devices may beentertainment equipment, audio equipment, portable electronic devices,navigation devices, weapons, automotive controls, and the like.

In embodiments, a body worn control device (e.g. as worn on a finger,attached to the hand at the palm, on the arm, leg, torso, and the like)may provide 3D position sensor information to the eyepiece. Forinstance, the control device may act as an ‘air mouse’, where 3Dposition sensors (e.g. accelerometers, gyros, and the like) provideposition information when a user commands so, such as with the click ofa button, a voice command, a visually detected gesture, and the like.The user may be able to use this feature to navigate either a 2D or 3Dimage being projected to the user via the eyepiece projection system.Further, the eyepiece may provide an external relay of the image fordisplay or projection to others, such as in the case of a presentation.The user may be able to change the mode of the control device between 2Dand 3D, in order to accommodate different functions, applications, userinterfaces, and the like. In embodiments, multiple 3D control devicesmay be utilized for certain applications, such as in simulationapplications.

In embodiments, a system may comprise an interactive head-mountedeyepiece worn by a user, wherein the eyepiece includes an opticalassembly through which the user views a surrounding environment anddisplayed content, wherein the optical assembly comprises a correctiveelement that corrects the user's view of the surrounding environment, anintegrated processor for handling content for display to the user, andan integrated image source for introducing the content to the opticalassembly; and a tactile control interface mounted on the eyepiece thataccepts control inputs from the user through at least one of a usertouching the interface and the user being proximate to the interface.

In embodiments, control of the eyepiece, and especially control of acursor associated with displayed content to the user, may be enabledthrough hand control, such as with a worn device 1500 as in FIG. 15, asa virtual computer mouse 1500A as in FIG. 15A, and the like. Forinstance, the worn device 1500 may transmit commands through physicalinterfaces (e.g. a button 1502, scroll wheel 1504), and the virtualcomputer mouse 1500A may be able interpret commands though detectingmotion and actions of the user's thumb, fist, hand, and the like. Incomputing, a physical mouse is a pointing device that functions bydetecting two-dimensional motion relative to its supporting surface. Aphysical mouse traditionally consists of an object held under one of theuser's hands, with one or more buttons. It sometimes features otherelements, such as “wheels”, which allow the user to perform varioussystem-dependent operations, or extra buttons or features that can addmore control or dimensional input. The mouse's motion translates intothe motion of a cursor on a display, which allows for fine control of agraphical user interface. In the case of the eyepiece, the user may beable to utilize a physical mouse, a virtual mouse, or combinations ofthe two. In embodiments, a virtual mouse may involve one or more sensorsattached to the user's hand, such as on the thumb 1502A, finger 1504A,palm 1508A, wrist 1510A, and the like, where the eyepiece receivessignals from the sensors and translates the received signals into motionof a cursor on the eyepiece display to the user. In embodiments, thesignals may be received through an exterior interface, such as thetactile interface 1402, through a receiver on the interior of theeyepiece, at a secondary communications interface, on an associatedphysical mouse or worn interface, and the like. The virtual mouse mayalso include actuators or other output type elements attached to theuser's hand, such as for haptic feedback to the user through vibration,force, pressure, electrical impulse, temperature, and the like. Sensorsand actuators may be attached to the user's hand by way of a wrap, ring,pad, glove, and the like. As such, the eyepiece virtual mouse may allowthe user to translate motions of the hand into motion of the cursor onthe eyepiece display, where ‘motions’ may include slow movements, rapidmotions, jerky motions, position, change in position, and the like, andmay allow users to work in three dimensions, without the need for aphysical surface, and including some or all of the six degrees offreedom. Note that because the ‘virtual mouse’ may be associated withmultiple portions of the hand, the virtual mouse may be implemented asmultiple ‘virtual mouse’ controllers, or as a distributed controlleracross multiple control members of the hand. In embodiments, theeyepiece may provide for the use of a plurality of virtual mice, such asfor one on each of the user's hands, one or more of the user's feet, andthe like.

In embodiments, the eyepiece virtual mouse may need no physical surfaceto operate, and detect motion such as through sensors, such as one of aplurality of accelerometer types (e.g. tuning fork, piezoelectric, shearmode, strain mode, capacitive, thermal, resistive, electromechanical,resonant, magnetic, optical, acoustic, laser, three dimensional, and thelike), and through the output signals of the sensor(s) determine thetranslational and angular displacement of the hand, or some portion ofthe hand. For instance, accelerometers may produce output signals ofmagnitudes proportional to the translational acceleration of the hand inthe three directions. Pairs of accelerometers may be configured todetect rotational accelerations of the hand or portions of the hand.Translational velocity and displacement of the hand or portions of thehand may be determined by integrating the accelerometer output signalsand the rotational velocity and displacement of the hand may bedetermined by integrating the difference between the output signals ofthe accelerometer pairs. Alternatively, other sensors may be utilized,such as ultrasound sensors, imagers, IR/RF, magnetometer, gyromagnetometer, and the like. As accelerometers, or other sensors, may bemounted on various portions of the hand, the eyepiece may be able todetect a plurality of movements of the hand, ranging from simple motionsnormally associated with computer mouse motion, to more highly complexmotion, such as interpretation of complex hand motions in a simulationapplication. In embodiments, the user may require only a smalltranslational or rotational action to have these actions translated tomotions associated with user intended actions on the eyepiece projectionto the user.

In embodiments, the virtual mouse may have physical switches associatedwith it to control the device, such as an on/off switch mounted on thehand, the eyepiece, or other part of the body. The virtual mouse mayalso have on/off control and the like through pre-defined motions oractions of the hand. For example, the operation of the virtual mouse maybe enabled through a rapid back and forth motion of the hand. In anotherexample, the virtual mouse may be disabled through a motion of the handpast the eyepiece, such as in front of the eyepiece. In embodiments, thevirtual mouse for the eyepiece may provide for the interpretation of aplurality of motions to operations normally associated with physicalmouse control, and as such, familiar to the user without training, suchas single clicking with a finger, double clicking, triple clicking,right clicking, left clicking, click and drag, combination clicking,roller wheel motion, and the like. In embodiments, the eyepiece mayprovide for gesture recognition, such as in interpreting hand gesturesvia mathematical algorithms.

In embodiments, gesture control recognition may be provided throughtechnologies that utilize capacitive changes resulting from changes inthe distance of a user's hand from a conductor element as part of theeyepiece's control system, and so would require no devices mounted onthe user's hand. In embodiments, the conductor may be mounted as part ofthe eyepiece, such as on the arm or other portion of the frame, or assome external interface mounted on the user's body or clothing. Forexample, the conductor may be an antenna, where the control systembehaves in a similar fashion to the touch-less musical instrument knownas the theremin. The theremin uses the heterodyne principle to generatean audio signal, but in the case of the eyepiece, the signal may be usedto generate a control input signal. The control circuitry may include anumber of radio frequency oscillators, such as where one oscillatoroperates at a fixed frequency and another controlled by the user's hand,where the distance from the hand varies the input at the controlantenna. In this technology, the user's hand acts as a grounded plate(the user's body being the connection to ground) of a variable capacitorin an L-C (inductance-capacitance) circuit, which is part of theoscillator and determines its frequency. In another example, the circuitmay use a single oscillator, two pairs of heterodyne oscillators, andthe like. In embodiments, there may be a plurality of differentconductors used as control inputs. In embodiments, this type of controlinterface may be ideal for control inputs that vary across a range, suchas a volume control, a zoom control, and the like. However, this type ofcontrol interface may also be used for more discrete control signals(e.g. on/off control) where a predetermined threshold determines thestate change of the control input.

In embodiments, the eyepiece may interface with a physical remotecontrol device, such as a wireless track pad mouse, hand held remotecontrol, body mounted remote control, remote control mounted on theeyepiece, and the like. The remote control device may be mounted on anexternal piece of equipment, such as for personal use, gaming,professional use, military use, and the like. For example, the remotecontrol may be mounted on a weapon for a soldier, such as mounted on apistol grip, on a muzzle shroud, on a fore grip, and the like, providingremote control to the soldier without the need to remove their handsfrom the weapon. The remote control may be removably mounted to theeyepiece.

In embodiments, a remote control for the eyepiece may be activatedand/or controlled through a proximity sensor. A proximity sensor may bea sensor able to detect the presence of nearby objects without anyphysical contact. For example, a proximity sensor may emit anelectromagnetic or electrostatic field, or a beam of electromagneticradiation (infrared, for instance), and look for changes in the field orreturn signal. The object being sensed is often referred to as theproximity sensor's target. Different proximity sensor targets may demanddifferent sensors. For example, a capacitive or photoelectric sensormight be suitable for a plastic target; an inductive proximity sensorrequires a metal target. Other examples of proximity sensor technologiesinclude capacitive displacement sensors, eddy-current, magnetic,photocell (reflective), laser, passive thermal infrared, passiveoptical, CCD, reflection of ionizing radiation, and the like. Inembodiments, the proximity sensor may be integral to any of the controlembodiments described herein, including physical remote controls,virtual mouse, interfaces mounted on the eyepiece, controls mounted onan external piece of equipment (e.g. a game controller, a weapon), andthe like.

In embodiments, sensors for measuring a user's body motion may be usedto control the eyepiece, or as an external input, such as using aninertial measurement unit (IMU), a 3-axis magnetometer, a 3-axis gyro, a3-axis accelerometer, and the like. For instance, an sensor may bemounted on the hand(s) of the user, thereby enabling the use of thesignals from the sensor for control the eyepiece, as described herein.In another instance, sensor signals may be received and interpreted bythe eyepiece to assess and/or utilize the body motions of the user forpurposes other than control. In an example, sensors mounted on each legand each arm of the user may provide signals to the eyepiece that allowthe eyepiece to measure the gait of the user. The gait of the user maythen in turn be used to monitor the gait of the user over time, such asto monitor changes in physical behavior, improvement during physicaltherapy, changes due to a head trauma, and the like. In the instance ofmonitoring for a head trauma, the eyepiece may initially determine abaseline gait profile for the user, and then monitor the user over time,such as before and after a physical event (e.g. a sports-relatedcollision, an explosion, an vehicle accident, and the like). In the caseof an athlete or person in physical therapy, the eyepiece may be usedperiodically to measure the gait of the user, and maintain themeasurements in a database for analysis. A running gait time profile maybe produced, such as to monitor the user's gait for indications ofphysical traumas, physical improvements, and the like.

In embodiments, control of the eyepiece, and especially control of acursor associated with displayed content to the user, may be enabledthrough the sensing of the motion of a facial feature, the tensing of afacial muscle, the clicking of the teeth, the motion of the jaw, and thelike, of the user wearing the eyepiece through a facial actuation sensor1502B. For instance, as shown in FIG. 15B, the eyepiece may have afacial actuation sensor as an extension from the eyepiece earphoneassembly 1504B, from the arm 1508B of the eyepiece, and the like, wherethe facial actuation sensor may sense a force, a vibration, and the likeassociated with the motion of a facial feature. The facial actuationsensor may also be mounted separate from the eyepiece assembly, such aspart of a standalone earpiece, where the sensor output of the earpieceand the facial actuation sensor may be either transferred to theeyepiece by either wired or wireless communication (e.g. Bluetooth orother communications protocol known to the art). The facial actuationsensor may also be attached to around the ear, in the mouth, on theface, on the neck, and the like. The facial actuation sensor may also becomprised of a plurality of sensors, such as to optimize the sensedmotion of different facial or interior motions or actions. Inembodiments, the facial actuation sensor may detect motions andinterpret them as commands, or the raw signals may be sent to theeyepiece for interpretation. Commands may be commands for the control ofeyepiece functions, controls associated with a cursor or pointer asprovided as part of the display of content to the user, and the like.For example, a user may click their teeth once or twice to indicate asingle or double click, such as normally associated with the click of acomputer mouse. In another example, the user may tense a facial muscleto indicate a command, such as a selection associated with the projectedimage. In embodiments, the facial actuation sensor may utilize noisereduction processing to minimize the background motions of the face, thehead, and the like, such as through adaptive signal processingtechnologies. A voice activity sensor may also be utilized to reduceinterference, such as from the user, from other individuals nearby, fromsurrounding environmental noise, and the like. In an example, the facialactuation sensor may also improve communications and eliminate noise bydetecting vibrations in the cheek of the user during speech, such aswith multiple microphones to identify the background noise and eliminateit through noise cancellation, volume augmentation, and the like.

In embodiments, the user of the eyepiece may be able to obtaininformation on some environmental feature, location, object, and thelike, viewed through the eyepiece by raising their hand into the fieldof view of the eyepiece and pointing at the object or position. Forinstance, the pointing finger of the user may indicate an environmentalfeature, where the finger is not only in the view of the eyepiece butalso in the view of an embedded camera. The system may now be able tocorrelate the position of the pointing finger with the location of theenvironmental feature as seen by the camera. Additionally, the eyepiecemay have position and orientation sensors, such as GPS and amagnetometer, to allow the system to know the location and line of sightof the user. From this, the system may be able to extrapolate theposition information of the environmental feature, such as to providethe location information to the user, to overlay the position of theenvironmental information onto a 2D or 3D map, to further associate theestablished position information to correlate that position informationto secondary information about that location (e.g. address, names ofindividuals at the address, name of a business at that location,coordinates of the location), and the like. Referring to FIG. 15C, in anexample, the user is looking though the eyepiece 1502C and pointing withtheir hand 1504C at a house 1508C in their field of view, where anembedded camera 1510C has both the pointed hand 1504C and the house1508C in its field of view. In this instance, the system is able todetermine the location of the house 1508C and provide locationinformation 1514C and a 3D map superimposed onto the user's view of theenvironment. In embodiments, the information associated with anenvironmental feature may be provided by an external facility, such ascommunicated with through a wireless communication connection, storedinternal to the eyepiece, such as downloaded to the eyepiece for thecurrent location, and the like. In embodiments, information provided tothe wearer of the eyepiece may include any of a plurality of informationrelated to the scene as viewed by the wearer, such as geographicinformation, point of interest information, social networkinginformation (e.g. Twitter, Facebook, and the like information related toa person standing in front of the wearer augmented around the person,such as ‘floating’ around the person), profile information (e.g. such asstored in the wearer's contact list), historical information, consumerinformation, product information, retail information, safetyinformation, advertisements, commerce information, security information,game related information, humorous annotations, news relatedinformation, and the like.

In embodiments, the user may be able to control their view perspectiverelative to a 3D projected image, such as a 3D projected imageassociated with the external environment, a 3D projected image that hasbeen stored and retrieved, a 3D displayed movie (such as downloaded forviewing), and the like. For instance, and referring again to FIG. 15C,the user may be able to change the view perspective of the 3D displayedimage 1512C, such as by turning their head, and where the live externalenvironment and the 3D displayed image stay together even as the userturns their head, moves their position, and the like. In this way, theeyepiece may be able to provide an augmented reality by overlayinginformation onto the user's viewed external environment, such as theoverlaid 3D displayed map 1512C, the location information 1514C, and thelike, where the displayed map, information, and the like, may change asthe user's view changes. In another instance, with 3D movies or 3Dconverted movies, the perspective of the viewer may be changed to putthe viewer ‘into’ the movie environment with some control of the viewingperspective, where the user may be able to move their head around andhave the view change in correspondence to the changed head position,where the user may be able to ‘walk into’ the image when they physicallywalk forward, have the perspective change as the user moves the gazingview of their eyes, and the like. In addition, additional imageinformation may be provided, such as at the sides of the user's viewthat could be accessed by turning the head.

In embodiments, the user of one eyepiece may be able to synchronizetheir view of a projected image with at least the view of a second userof an eyepiece. For instance, two separate eyepiece users may wish toview the same 3D map, game projection, point-of-interest projection, andthe like, where the two viewers are not only seeing the same projectedcontent, but where the projected content's view is synchronized betweenthem. In an example, two users may want to jointly view a 3D map of aregion, and the image is synchronized such that the one user may be ableto point at a position on the 3D map that the other user is able to seeand interact with. The two users may be able to move around the 3D mapand share a virtual-physical interaction between the two users and the3D map, and the like. Further, a group of eyepiece wearers may be ableto jointly interact with a projection as a group. In this way, two ormore users may be able to have a unified augmented reality experiencethrough the coordination-synchronization of their eyepieces.Synchronization of two or more eyepieces may be provided bycommunication of position information between the eyepieces, such asabsolute position information, relative position information,translation and rotational position information, and the like, such asfrom position sensors as described herein (e.g. gyroscopes, IMU, GPS,and the like). Communications between the eyepieces may be direct,through an Internet network, through the cell-network, through asatellite network, and the like. Processing of position informationcontributing to the synchronization may be executed in a masterprocessor in a single eyepiece, collectively amongst a group ofeyepieces, in remote server system, and the like, or any combinationthereof. In embodiments, the coordinated, synchronized view of projectedcontent between multiple eyepieces may provide an extended augmentedreality experience from the individual to a plurality of individuals,where the plurality of individuals benefit from the group augmentedreality experience. For example, a group of concertgoers may synchronizetheir eyepieces with a feed from the concert producers such that visualeffects or audio may be pushed to people with eyepieces by the concertproducer, performers, other audience members, and the like. In anexample, the performer may have a master eyepiece and may controlsending content to audience members. In one embodiment, the content maybe the performer's view of the surrounding environment. The performermay be using the master eyepiece for applications as well, such ascontrolling an external lighting system, interacting with an augmentedreality drum kit or sampling board, calling up song lyrics, and thelike.

In embodiments, the eyepiece may utilize sound projection techniques torealize a direction of sound for the wearer of the eyepiece, such aswith surround sound techniques. Realization of a direction of sound fora wearer may include the reproduction of the sound from the direction oforigin, either in real-time or as a playback. It may include a visual oraudible indicator to provide a direction for the source of sound. Soundprojection techniques may be useful to an individual that has theirhearing impaired or blocked, such as due to the user experiencinghearing loss, a user wearing headphones, a user wearing hearingprotection, and the like. In this instance, the eyepiece may provideenhanced 3D audible reproduction. In an example, the wearer may haveheadphones on, and a gunshot has been fired. In this example, theeyepiece may be able to reproduce the 3D sound profile for the sound ofthe gunshot, thus allowing the wearer to respond to the gunshot knowingwhere the sound came from. In another example, a wearer with headphones,hearing loss, in a loud environment, and the like, may not otherwise beable to tell what's being said and/or the direction of the personspeaking, but is provided with a 3D sound enhancement from the eyepiece(e.g. the wearer is listening to other proximate individuals throughheadphones and so does not have directionality information). In anotherexample, a wearer may be in a loud ambient environment, or in anenvironment where periodic loud noises can occur. In this instance, theeyepiece may have the ability to cut off the loud sound to protect thewearer's hearing, or the sound could be so loud that the wearer can'ttell where the sound came from, and further, now their ears could beringing so loud they can't hear anything. To aid in this situation, theeyepiece may provide visible, auditory, vibration, and the like queuesto the wearer to indicate the direction of the sound source. Inembodiments, the eyepiece may provide “augmented” hearing where thewearer's ears are plugged to protect their ears from loud noises, butusing the ear buds to generate a reproduction of sound to replace what'smissing form the natural world. This artificial sound may then be usedto give directionality to wirelessly transmitted communication that theoperator couldn't hear naturally.

In embodiments, an example of a configuration for establishingdirectionality of a source sound may be point different microphones indifferent directions. For instance, at least one microphone may be usedfor the voice of the wearer, at least one microphone for the surroundingenvironment, at least one pointing down at the ground, and potentiallyin a plurality of different discrete directions. In this instance, themicrophone pointing down may be subtracted to isolate other sounds,which may be combined with 3D sound surround, and augmented hearingtechniques, as described herein.

In an example of a sound augmented system as part of the eyepiece, thereare a number of users with eyepieces, such as in a noisy environmentwhere all the users have ‘plugged ears’ as implemented throughartificial noise blockage through the eyepiece ear buds. One of wearersmay yell out that they need some piece of equipment. Because of all theambient noise and the hearing protection the eyepiece creates, no onecan hear the request for equipment. Here, the wearer making the verbalrequest has a filtered microphone close to their mouth, and they couldwirelessly transmit the request to the others, where their eyepiececould relay a sound signal to the other user's eyepieces, and to the earon the correct side, and the others would know to look to the right orleft to see who has made the request. This system could be furtherenhanced with geo-locations of all the wearers, and a “virtual” surroundsound system that uses the two ear buds to give the perception of 3Dspace (such as the SRS True Surround Technology).

In embodiments, auditory queues could also be computer generated so thecommunicating user doesn't need to verbalize their communication but canselect it from a list of common commands, the computer generates thecommunication based on preconfigured conditions, and the like. In anexample, the wearers may be in a situation where they don't want adisplay in front of their eyes but want to have ear buds in their ears.In this case, if they wanted to notify someone in a group to get up andfollow them, they could just click a controller a certain number oftimes, or provide a visual hand gesturer with a camera, an IMU, and thelike. The system may choose the ‘follow me’ command and transmit it tothe other users with the communicating user's location for the 3D systemto trick them into hearing from where they are actually sitting out ofsight of them. In embodiments, directional information may be determinedand/or provided through position information from the users ofeyepieces.

In embodiments, the eyepiece may provide aspects of signals intelligence(SIGINT), such as in the use of existing WiFi, 3G, Bluetooth, and thelike communications signals to gather signals intelligence for devicesand users in proximity to the wearer of the eyepiece. These signals maybe from other eyepieces, such as to gather information about other knownfriendly users; other eyepieces that have been picked up by anunauthorized individual, such as through a signal that is generated whenan unauthorized user tries to use the eyepiece; other communicationsdevices (e.g. radios, cell phones, pagers, walky-talkies, and the like);electronic signals emanating from devices that may not be directly usedfor communications; and the like. Information gathered by the eyepiecemay be direction information, position information, motion information,number of and/or rate of communications, and the like. Further,information may be gathered through the coordinated operations ofmultiple eyepieces, such as in the triangulation of a signal fordetermination of the signal's location.

Referring to FIG. 15D, in embodiments the user of the eyepiece 1502D maybe able to use multiple hand/finger points from their hand 1504D todefine the field of view (FOV) 1508D of the camera 1510D relative to thesee-thru view, such as for augmented reality applications. For instance,in the example shown, the user is utilizing their first finger and thumbto adjust the FOV 1508D of the camera 1510D of the eyepiece 1502D. Theuser may utilize other combinations to adjust the FOV 1508D, such aswith combinations of fingers, fingers and thumb, combinations of fingersand thumbs from both hands, use of the palm(s), cupped hand(s), and thelike. The use of multiple hand/finger points may enable the user toalter the FOV 1508 of the camera 1510D in much the same way as users oftouch screens, where different points of the hand/finger establishpoints of the FOV to establish the desired view. In this instancehowever, there is no physical contact made between the user's hand(s)and the eyepiece. Here, the camera may be commanded to associateportions of the user's hand(s) to the establishing or changing of theFOV of the camera. The command may be any command type described herein,including and not limited to hand motions in the FOV of the camera,commands associated with physical interfaces on the eyepiece, commandsassociated with sensed motions near the eyepiece, commands received froma command interface on some portion of the user, and the like. Theeyepiece may be able to recognize the finger/hand motions as thecommand, such as in some repetitive motion. In embodiments, the user mayalso utilize this technique to adjust some portion of the projectedimage, where the eyepiece relates the viewed image by the camera to someaspect of the projected image, such as the hand/finger points in view tothe projected image of the user. For example, the user may besimultaneously viewing the external environment and a projected image,and the user utilizes this technique to change the projected viewingarea, region, magnification, and the like. In embodiments, the user mayperform a change of FOV for a plurality of reasons, including zooming inor out from a viewed scene in the live environment, zoom in or out froma viewed portion of the projected image, to change the viewing areaallocated to the projected image, to change the perspective view of theenvironment or projected image, and the like.

In embodiments, the eyepiece may enable simultaneous FOVs. For example,simultaneous wide, medium, and narrow camera FOVs may be used, where theuser can have different FOVs up simultaneously in view (i.e. wide toshow the entire field, perhaps static, and narrow to focus on aparticular target, perhaps moving with the eye or with a cursor).

In embodiments the eyepiece may be able to determine where the user isgazing, or the motion of the user's eye, by tracking the eye throughreflected light off the user's eye. This information may then be used tohelp correlate the user's line of sight with respect to the projectedimage, a camera view, the external environment, and the like, and usedin control techniques as described herein. For instance, the user maygaze at a location on the projected image and make a selection, such aswith an external remote control or with some detected eye movement (e.g.blinking). In an example of this technique, and referring to FIG. 15E,transmitted light 1508E, such as infrared light, may be reflected 1510Efrom the eye 1504E and sensed at the optical display 502 (e.g. with acamera or other optical sensor). The information may then be analyzed toextract eye rotation from changes in reflections. In embodiments, an eyetracking facility may use the corneal reflection and the center of thepupil as features to track over time; use reflections from the front ofthe cornea and the back of the lens as features to track; image featuresfrom inside the eye, such as the retinal blood vessels, and follow thesefeatures as the eye rotates; and the like. Alternatively, the eyepiecemay use other techniques to track the motions of the eye, such as withcomponents surrounding the eye, mounted in contact lenses on the eye,and the like. For instance, a special contact lens may be provided tothe user with an embedded optical component, such as a mirror, magneticfield sensor, and the like, for measuring the motion of the eye. Inanother instance, electric potentials may be measured and monitored withelectrodes placed around the eyes, utilizing the steady electricpotential field from the eye as a dipole, such as with its positive poleat the cornea and its negative pole at the retina. In this instance, theelectric signal may be derived using contact electrodes placed on theskin around the eye, on the frame of the eyepiece, and the like. If theeye moves from the centre position towards the periphery, the retinaapproaches one electrode while the cornea approaches the opposing one.This change in the orientation of the dipole and consequently theelectric potential field results in a change in the measured signal. Byanalyzing these changes eye movement may be tracked.

In another example of how eye gaze direction of the user and associatedcontrol may be applied involves placement (by the eyepiece) and optionalselection (by the user) of a visual indicator in the user's peripheralvision, such as in order to reduce clutter in the narrow portion of theuser's visual field around the gaze direction where the eye's highestvisual input resides. Since the brain is limited as to how muchinformation it can process at a time, and the brain pays the mostattention to visual content close to the direction of gaze, the eyepiecemay provide projected visual indicators in the periphery of vision ascues to the user. This way the brain may only have to process thedetection of the indicator, and not the information associated with theindicator, thus decrease the potential for overloading the user withinformation. The indicator may be an icon, a picture, a color, symbol, ablinking object, and the like, and indicate an alert, an email arriving,an incoming phone call, a calendar event, an internal or externalprocessing facility that requires attention from the user, and the like.With the visual indicator in the periphery, the user may become aware ofit without being distracted by it. The user may then optionally decideto elevate the content associated with the visual cue in order to seemore information, such as gazing over to the visual indicator, and bydoing so, opening up it's content. For example, an icon representing anincoming email may indicate an email being received. The user may noticethe icon, and choose to ignore it (such as the icon disappearing after aperiod of time if not activated, such as by a gaze or some other controlfacility). Alternately, the user may notice the visual indicator andchoose to ‘active’ it by gazing in the direction of the visualindicator. In the case of the email, when the eyepiece detects that theuser's eye gaze is coincident with the location of the icon, theeyepiece may open up the email and reveal it's content. In this way theuser maintains control over what information is being paid attention to,and as a result, minimize distractions and maximize content usageefficiency.

In embodiments, the eyepiece may utilize sub-conscious control aspects,such as images in the wearer's periphery, images presented to the userat rates below conscious perception, sub-conscious perceptions to aviewed scene by the viewer, and the like. For instance, a wearer may bepresented images through the eyepiece that are at a rate the wearer isunaware of, but is subconsciously made aware of as presented content,such as a reminder, an alert (e.g. an alert that calls on the wearer toincrease a level of attention to something, but not so much so that theuser needs a full conscious reminder), an indication related to thewearer's immediate environment (e.g. the eyepiece has detected somethingin the wearer's field of view that may have some interest to the wearer,and to which the indication draws the wearer's attention), and the like.In another instance, the eyepiece may provide indicators to the wearerthrough a brain activity monitoring interface, where electrical signalswithin the brain fire before a person realizes they've recognized animage. For instance, the brain activity-monitoring interface may includeelectroencephalogram (EEG) sensors (or the like) to monitor brainactivity as the wearer is viewing the current environment. When theeyepiece, through the brain activity-monitoring interface, senses thatthe wearer has become ‘aware’ of an element of the surroundingenvironment, the eyepiece may provide conscious level feedback to thewearer to make the wearer more aware of the element. For example, awearer may unconsciously become aware of seeing a familiar face in acrowd (e.g. a friend, a suspect, a celebrity), and the eyepiece providesa visual or audio indication to the wearer to bring the person moreconsciously to the attention of the wearer. In another example, thewearer may view a product that arouses their attention at a subconsciouslevel, and the eyepiece provides a conscious indication to the wearer,more information about the product, an enhanced view of the product, alink to more information about the product, and the like. Inembodiments, the ability for the eyepiece to extend the wearer's realityto a subconscious level may enable the eyepiece to provide the wearerwith an augmented reality beyond their normal conscious experience withthe world around them.

In embodiments, the eyepiece may have a plurality of modes of operationwhere control of the eyepiece is controlled at least in part bypositions, shapes, motions of the hand, and the like. To provide thiscontrol the eyepiece may utilize hand recognition algorithms to detectthe shape of the hand/fingers, and to then associate those handconfigurations, possibly in combination with motions of the hand, ascommands. Realistically, as there may be only a limited number of handconfigurations and motions available to command the eyepiece, these handconfigurations may need to be reused depending upon the mode ofoperation of the eyepiece. In embodiments, certain hand configurationsor motions may be assigned for transitioning the eyepiece from one modeto the next, thereby allowing for the reuse of hand motions. Forinstance, and referring to FIG. 15F, the user's hand 1504F may be movedin view of a camera on the eyepiece, and the movement may then beinterpreted as a different command depending upon the mode, such as acircular motion 1508F, a motion across the field of view 1510F, a backand forth motion 1512F, and the like. In a simplistic example, supposethere are two modes of operation, mode one for panning a view from theprojected image and mode two for zooming the projected image. In thisexample the user may want to use a left-to-right finger-pointed handmotion to command a panning motion to the right. However, the user mayalso want to use a left-to-right finger-pointed hand motion to command azooming of the image to greater magnification. To allow the dual use ofthis hand motion for both command types, the eyepiece may be configuredto interpret the hand motion differently depending upon the mode theeyepiece is currently in, and where specific hand motions have beenassigned for mode transitions. For instance, a clockwise rotationalmotion may indicate a transition from pan to zoom mode, and acounter-clockwise rotational motion may indicate a transition from zoomto pan mode. This example is meant to be illustrative and not limitingin anyway, where one skilled in the art will recognize how this generaltechnique could be used to implement a variety of command/modestructures using the hand(s) and finger(s), such as hand-fingerconfigurations-motions, two-hand configuration-motions, and the like.

In embodiments, a system may comprise an interactive head-mountedeyepiece worn by a user, wherein the eyepiece includes an opticalassembly through which the user views a surrounding environment anddisplayed content, wherein the optical assembly comprises a correctiveelement that corrects the user's view of the surrounding environment, anintegrated processor for handling content for display to the user, andan integrated image source for introducing the content to the opticalassembly; and an integrated camera facility that images a gesture,wherein the integrated processor identifies and interprets the gestureas a command instruction. The control instruction may providemanipulation of the content for display, a command communicated to anexternal device, and the like.

In embodiments, control of the eyepiece may be enabled through eyemovement, an action of the eye, and the like. For instance, there may bea camera on the eyepiece that views back to the wearer's eye(s), whereeye movements or actions may be interpreted as command information, suchas through blinking, repetitive blinking, blink count, blink rate, eyeopen-closed, gaze tracking, eye movements to the side, up and down, sideto side, through a sequence of positions, to a specific position, dwelltime in a position, gazing toward a fixed object (e.g. the corner of thelens of the eyepiece), through a certain portion of the lens, at areal-world object, and the like. In addition, eye control may enable theviewer to focus on a certain point on the displayed image from theeyepiece, and because the camera may be able to correlate the viewingdirection of the eye to a point on the display, the eyepiece may be ableto interpret commands through a combination of where the wearer islooking and an action by the wearer (e.g. blinking, touching aninterface device, movement of a position sense device, and the like).For example, the viewer may be able to look at an object on the display,and select that object through the motion of a finger enabled through aposition sense device.

In some embodiments, the glasses may be equipped with eye trackingdevices for tracking movement of the user's eye, or preferably botheyes; alternatively, the glasses may be equipped with sensors forsix-degree freedom of movement tracking, i.e., head movement tracking.These devices or sensors are available, for example, from Chronos VisionGmbH, Berlin, Germany and ISCAN, Woburn, Mass. Retinal scanners are alsoavailable for tracking eye movement. Retinal scanners may also bemounted in the augmented reality glasses and are available from avariety of companies, such as Tobii, Stockholm, Sweden, and SMI, Teltow,Germany, and ISCAN.

The augmented reality eyepiece also includes a user input interface, asshown, to allow a user to control the device. Inputs used to control thedevice may include any of the sensors discussed above, and may alsoinclude a trackpad, one or more function keys and any other suitablelocal or remote device. For example, an eye tracking device may be usedto control another device, such as a video game or external trackingdevice. As an example, FIG. 29A depicts a user with an augmented realityeyepiece equipped with an eye tracking device 2900A, discussed elsewherein this document. The eye tracking device allows the eyepiece to trackthe direction of the user's eye or preferably, eyes, and send themovements to the controller of the eyepiece. Control system includes theaugmented reality eyepiece and a control device for the weapon. Themovements may then be transmitted to the control device for a weaponcontrolled by the control device, which may be within sight of the user.The movement of the user's eyes is then converted by suitable softwareto signals for controlling movement in the weapon, such as quadrant(range) and azimuth (direction). Additional controls may be used inconjunction with eye tracking, such as with the user's trackpad orfunction keys. The weapon may be large caliber, such as a howitzer ormortar, or may small caliber, such as a machine gun.

The movement of the user's eyes is then converted by suitable softwareto signals for controlling movement of the weapon, such as quadrant(range) and azimuth (direction) of the weapon. Additional controls maybe used for single or continuous discharges of the weapon, such as withthe user's trackpad or function keys. Alternatively, the weapon may bestationary and non-directional, such as an implanted mine orshape-charge, and may be protected by safety devices, such as byrequiring specific encoded commands. The user of the augmented realitydevice may activate the weapon by transmitting the appropriate codes andcommands, without using eye-tracking features.

In embodiments, control of the eyepiece may be enabled though gesturesby the wearer. For instance, the eyepiece may have a camera that viewsoutward (e.g. forward, to the side, down) and interprets gestures ormovements of the hand of the wearer as control signals. Hand signals mayinclude passing the hand past the camera, hand positions or signlanguage in front of the camera, pointing to a real-world object (suchas to activate augmentation of the object), and the like. Hand motionsmay also be used to manipulate objects displayed on the inside of thetranslucent lens, such as moving an object, rotating an object, deletingan object, opening-closing a screen or window in the image, and thelike. Although hand motions have been used in the preceding examples,any portion of the body or object held or worn by the wearer may also beutilized for gesture recognition by the eyepiece.

In embodiments, head motion control may be used to send commands to theeyepiece, where motion sensors such as accelerometers, gyros, or anyother sensor described herein, may be mounted on the wearer's head, onthe eyepiece, in a hat, in a helmet, and the like. Referring to FIG.14A, head motions may include quick motions of the head, such as jerkingthe head in a forward and/or backward motion 1412, in an up and/or downmotion 1410, in a side to side motion as a nod, dwelling in a position,such as to the side, moving and holding in position, and the like.Motion sensors may be integrated into the eyepiece, mounted on theuser's head or in a head covering (e.g. hat, helmet) by wired orwireless connection to the eyepiece, and the like. In embodiments, theuser may wear the interactive head-mounted eyepiece, where the eyepieceincludes an optical assembly through which the user views a surroundingenvironment and displayed content. The optical assembly may include acorrective element that corrects the user's view of the surroundingenvironment, an integrated processor for handling content for display tothe user, and an integrated image source for introducing the content tothe optical assembly. At least one of a plurality of head motion sensingcontrol devices may be integrated or in association with the eyepiecethat provide control commands to the processor as command instructionsbased upon sensing a predefined head motion characteristic. The headmotion characteristic may be a nod of the user's head such that the nodis an overt motion dissimilar from ordinary head motions. The overtmotion may be a jerking motion of the head. The control instructions mayprovide manipulation of the content for display, be communicated tocontrol an external device, and the like. Head motion control may beused in combination with other control mechanisms, such as using anothercontrol mechanism as discussed herein to activate a command and for thehead motion to execute it. For example, a wearer may want to move anobject to the right, and through eye control, as discussed herein,select the object and activate head motion control. Then, by tippingtheir head to the right, the object may be commanded to move to theright, and the command terminated through eye control.

In embodiments, the eyepiece may be controlled through audio, such asthrough a microphone. Audio signals may include speech recognition,voice recognition, sound recognition, sound detection, and the like.Audio may be detected though a microphone on the eyepiece, a throatmicrophone, a jaw bone microphone, a boom microphone, a headphone, earbud with microphone, and the like.

In embodiments, command inputs may provide for a plurality of controlfunctions, such as turning on/off the eyepiece projector, turn on/offaudio, turn on/off a camera, turn on/off augmented reality projection,turn on/off GPS, interaction with display (e.g. select/accept functiondisplayed, replay of captured image or video, and the like), interactionwith the real-world (e.g. capture image or video, turn a page of adisplayed book, and the like), perform actions with an embedded orexternal mobile device (e.g. mobile phone, navigation device, musicdevice, VoIP, and the like), browser controls for the Internet (e.g.submit, next result, and the like), email controls (e.g. read email,display text, text-to-speech, compose, select, and the like), GPS andnavigation controls (e.g. save position, recall saved position, showdirections, view location on map), and the like.

In embodiments, the eyepiece may provide 3D display imaging to the user,such as through conveying a stereoscopic, auto-stereoscopic,computer-generated holography, volumetric display image,stereograms/stereoscopes, view-sequential displays, electro-holographicdisplays, parallax “two view” displays and parallax panoramagrams,re-imaging systems, and the like, creating the perception of 3D depth tothe viewer. Display of 3D images to the user may employ different imagespresented to the user's left and right eyes, such as where the left andright optical paths have some optical component that differentiates theimage, where the projector facility is projecting different images tothe user's left and right eye's, and the like. The optical path,including from the projector facility through the optical path to theuser's eye, may include a graphical display device that forms a visualrepresentation of an object in three physical dimensions. A processor,such as the integrated processor in the eyepiece or one in an externalfacility, may provide 3D image processing as at least a step in thegeneration of the 3D image to the user.

In embodiments, holographic projection technologies may be employed inthe presentation of a 3D imaging effect to the user, such ascomputer-generated holography (CGH), a method of digitally generatingholographic interference patterns. For instance, a holographic image maybe projected by a holographic 3D display, such as a display thatoperates on the basis of interference of coherent light. Computergenerated holograms have the advantage that the objects which one wantsto show do not have to possess any physical reality at all, that is,they may be completely generated as a ‘synthetic hologram’. There are aplurality of different methods for calculating the interference patternfor a CGH, including from the fields of holographic information andcomputational reduction as well as in computational and quantizationtechniques. For instance, the Fourier transform method and point sourceholograms are two examples of computational techniques. The Fouriertransformation method may be used to simulate the propagation of eachplane of depth of the object to the hologram plane, where thereconstruction of the image may occur in the far field. In an exampleprocess, there may be two steps, where first the light field in the farobserver plane is calculated, and then the field is Fourier transformedback to the lens plane, where the wavefront to be reconstructed by thehologram is the superposition of the Fourier transforms of each objectplane in depth. In another example, a target image may be multiplied bya phase pattern to which an inverse Fourier transform is applied.Intermediate holograms may then be generated by shifting this imageproduct, and combined to create a final set. The final set of hologramsmay then be approximated to form kinoforms for sequential display to theuser, where the kinoform is a phase hologram in which the phasemodulation of the object wavefront is recorded as a surface-reliefprofile. In the point source hologram method the object is broken downin self-luminous points, where an elementary hologram is calculated forevery point source and the final hologram is synthesized bysuperimposing all the elementary holograms.

In an embodiment, 3-D or holographic imagery may be enabled by a dualprojector system where two projectors are stacked on top of each otherfor a 3D image output. Holographic projection mode may be entered by acontrol mechanism described herein or by capture of an image or signal,such as an outstretched hand with palm up, an SKU, an RFID reading, andthe like. For example, a wearer of the eyepiece may view a letter ‘X’ ona piece of cardboard which causes the eyepiece to enter holographic modeand turning on the second, stacked projector. Selecting what hologram todisplay may be done with a control technique. The projector may projectthe hologram onto the cardboard over the letter ‘X’. Associated softwaremay track the position of the letter ‘X’ and move the projected imagealong with the movement of the letter ‘X’. In another example, theeyepiece may scan a SKU, such as a SKU on a toy construction kit, and a3-D image of the completed toy construction may be accessed from anonline source or non-volatile memory. Interaction with the hologram,such as rotating it, zooming in/out, and the like, may be done using thecontrol mechanisms described herein. Scanning may be enabled byassociated bar code/SKU scanning software. In another example, akeyboard may be projected in space or on a surface. The holographickeyboard may be used in or to control any of the associatedapplications/functions.

In embodiments, eyepiece facilities may provide for locking the positionof a virtual keyboard down relative to a real environmental object (e.g.a table, a wall, a vehicle dashboard, and the like) where the virtualkeyboard then does not move as the wearer moves their head. In anexample, and referring to FIG. 24, the user may be sitting at a tableand wearing the eyepiece 2402, and wish to input text into anapplication, such as a word processing application, a web browser, acommunications application, and the like. The user may be able to bringup a virtual keyboard 2408, or other interactive control element (e.g.virtual mouse, calculator, touch screen, and the like), to use forinput. The user may provide a command for bringing up the virtualkeyboard 2408, and use a hand gesture 2404 for indicating the fixedlocation of the virtual keyboard 2408. The virtual keyboard 2408 maythen remain fixed in space relative to the outside environment, such asfixed to a location on the table 2410, where the eyepiece facilitieskeep the location of the virtual keyboard 2408 on the table 2410 evenwhen the user turns their head. That is, the eyepiece 2402 maycompensate for the user's head motion in order to keep the user's viewof the virtual keyboard 2408 located on the table 2410. In embodiments,the user may wear the interactive head-mounted eyepiece, where theeyepiece includes an optical assembly through which the user views asurrounding environment and displayed content. The optical assembly mayinclude a corrective element that corrects the user's view of thesurrounding environment, an integrated processor for handling contentfor display to the user, and an integrated image source for introducingthe content to the optical assembly. An integrated camera facility maybe provided that images the surrounding environment, and identifies auser hand gesture as an interactive control element location command,such as a hand-finger configuration moved in a certain way, positionedin a certain way, and the like. The location of the interactive controlelement then may remain fixed in position with respect to an object inthe surrounding environment, in response to the interactive controlelement location command, regardless of a change in the viewingdirection of the user. In this way, the user may be able to utilize avirtual keyboard in much the same way they would a physical keyboard,where the virtual keyboard remains in the same location. However, in thecase of the virtual keyboard there are not ‘physical limitations’, suchas gravity, to limit where the user may locate the keyboard. Forinstance, the user could be standing next to a wall, and place thekeyboard location on the wall, and the like. It will be appreciated byone skilled in the art that the ‘virtual keyboard’ technology may beapplied to any controller, such as a virtual mouse, virtual touch pad,virtual game interface, virtual phone, virtual calculator, virtualpaintbrush, virtual drawing pad, and the like. For example, a virtualtouchpad may be visualized through the eyepiece to the user, andpositioned by the user such as by use of hand gestures, and used inplace of a physical touchpad.

In embodiments, eyepiece facilities may use visual techniques to renderthe projection of an object (e.g. virtual keyboard, keypad, calculator,notepad, joystick, control panel, book) onto a surface, such as byapplying distortions like parallax, keystone, and the like. For example,the appearance of a keyboard projected onto a tabletop in front of theuser with proper perspective may be aided through applying a keystoneeffect, where the projection as provided through the eyepiece to theuser is distorted so that it looks like it is lying down on the surfaceof the table. In addition, these techniques may be applied dynamically,to provide the proper perspective even as the user moves around inrelationship to the surface.

In embodiments, eyepiece facilities may provide for gesture recognitionthat may be used to provide a keyboard and mouse experience with theeyepiece. For instance, with images of a keyboard, mouse, and fingersoverlaid on the lower part of the display, the system may be capable oftracking finger positions in real time to enable a virtual desktop.Through gesture recognition, tracking may be done without wires andexternal powered devices. In another instance, fingertip locations maybe tracked through gesture recognition through the eyepiece withoutwires and external power, such as with gloves with passive RFID chips ineach fingertip. In this instance, each RFID chip may have its ownresponse characteristic, enabling a plurality of digits of the fingersto be read simultaneously. The RFID chips may be paired with the eyewearso that they are distinguishable from other RFID chips that may beoperating nearby. The eyewear may provide the signals to activate theRFID chips and have two or more receiving antennas. Each receivingantenna may be connected to a phase-measurement circuit element that inturn provides input to a location-determining algorithm. Thelocation-determining algorithm may also provide velocity andacceleration information, and the algorithm that ultimately may providekeyboard and mouse information to the eyepiece operating system. Inembodiments, with two receiving antennas, the azimuthal positions ofeach fingertip can be determined with the phase difference between thereceiving antennas. The relative phase difference between RFID chips maythen be used to determine the radial positions of the fingertips.

In embodiments, eyepiece facilities may use visual techniques to renderthe projection of a previously taken medical scan onto the wearer'sbody, such as an x-ray, an ultrasound, an MRI, a PET scan, and the like.For example, and referring to FIG. 24A, the eyepiece may have access toan x-ray image taken of the wearer's hand. The eyepiece may then utilizeits integrated camera to view the wear's hand 2402A, and overlay aprojected image 2404A of the x-ray onto the hand. Further, the eyepiecemay be able to maintain the image overlay as the wearer moves their handand gaze relative to one other. In embodiments, this technique may alsobe implemented while the wearer is looking in the mirror, where theeyepiece transposes an image on top of the reflected image. Thistechnique may be used as part of a diagnostic procedure, forrehabilitation during physical therapy, to encourage exercise and diet,to explain to a patient a diagnosis or condition, and the like. Theimages may be the images of the wearer, generic images from a databaseof images for medical conditions, and the like. The generic overlay mayshow some type of internal issue that is typical of a physicalcondition, a projection of what the body will look like if a certainroutine is followed for a period of time, and the like. In embodiments,an external control device, such as pointer controller, may enable themanipulation of the image. Further, the overlay of the image may besynchronized between multiple people, each wearing an eyepiece, asdescribed herein. For instance, a patient and a doctor may both projectthe image onto the patient's hand, where the doctor may now explain aphysical ailment while the patient views the synchronized images of theprojected scan and the doctor's explanation.

In embodiments, eyepiece facilities may provide for removing theportions of a virtual keyboard projection where intervening obstructionsappear (e.g. the user's hand getting in the way, where it is not desiredto project the keyboard onto the user's hand). In an example, andreferring to FIG. 30, the eyepiece 3002 may provide a projected virtualkeyboard 3008 to the wearer, such as onto a tabletop. The wearer maythen reach ‘over’ the virtual keyboard 3008 to type. As the keyboard ismerely a projected virtual keyboard, rather than a physical keyboard,without some sort of compensation to the projected image the projectedvirtual computer would be projected ‘onto’ the back of the user's hand.However, as in this example, the eyepiece may provide compensation tothe projected image such that the portion of the wearer's hand 3004 thatis obstructing the intended projection of the virtual keyboard onto thetable may be removed from the projection. That is, it may not bedesirable for portions of the keyboard projection 3008 to be visualizedonto the user's hand, and so the eyepiece subtracts the portion of thevirtual keyboard projection that is co-located with the wearer's hand3004. In embodiments, the user may wear the interactive head-mountedeyepiece, where the eyepiece includes an optical assembly through whichthe user views a surrounding environment and displayed content. Theoptical assembly may include a corrective element that corrects theuser's view of the surrounding environment, an integrated processor forhandling content for display to the user, and an integrated image sourcefor introducing the content to the optical assembly. The displayedcontent may include an interactive control element (e.g. virtualkeyboard, virtual mouse, calculator, touch screen, and the like). Anintegrated camera facility may image a user's body part as it interactswith the interactive control element, wherein the processor removes aportion of the interactive control element by subtracting the portion ofthe interactive control element that is determined to be co-located withthe imaged user body part based on the user's view. In embodiments, thistechnique of partial projected image removal may be applied to otherprojected images and obstructions, and is not meant to be restricted tothis example of a hand over a virtual keyboard.

In embodiments, eyepiece facilities may provide for interveningobstructions for any virtual content that is displayed over “real” worldcontent. If some reference frame is determined that places the contentat some distance, then any object that passes between the virtual imageand the viewer may be subtracted from the displayed content so as not tocreate a discontinuity for the user that is expecting the displayedinformation to exist at a certain distance away. In embodiments,variable focus techniques may also be used to increase the perception ofa distance hierarchy amongst the viewed content.

In embodiments, eyepiece facilities may provide for the ability todetermine an intended text input from a sequence of character contactsswiped across a virtual keypad, such as with the finger, a stylus, theentire hand, and the like. For example, and referring to FIG. 37, theeyepiece may be projecting a virtual keyboard 3700, where the userwishes to input the word ‘wind’. Normally, the user would discretelypress the key positions for ‘w’, then ‘i’, then ‘n’, and finally ‘d’,and a facility (camera, accelerometer, and the like, such as describedherein) associated with the eyepiece would interpret each position asbeing the letter for that position. However, the system may also be ableto monitor the movement, or swipe, of the user's finger or otherpointing device across the virtual keyboard and determine best fitmatches for the pointer movement. In the figure, the pointer has startedat the character ‘w’ and swept a path 3704 though the characters e, r,t, y, u, i, k, n, b, v, f, and d where it stops. The eyepiece mayobserve this sequence and determine the sequence, such as through aninput path analyzer, feed the sensed sequence into a word matchingsearch facility, and output a best fit word, in this case ‘wind’ as text3708. In embodiments, the eyepiece may monitor the motion of thepointing device across the keypad and determine the word more directly,such as though auto complete word matching, pattern recognition, objectrecognition, and the like, where some ‘separator’ indicates the spacebetween words, such as a pause in the motion of the pointing device, atap of the pointing device, a swirling motion of the pointing device,and the like. For instance, the entire swipe path may be used withpattern or object recognition algorithms to associate whole words withthe discrete patterns formed by the user's finger as they move througheach character to form words, with a pause between the movements asdemarcations between the words. The eyepiece may provide the best-fitword, a listing of best-fit words, and the like. In embodiments, theuser may wear the interactive head-mounted eyepiece, where the eyepieceincludes an optical assembly through which the user views a surroundingenvironment and displayed content. The optical assembly may include acorrective element that corrects the user's view of the surroundingenvironment, an integrated processor for handling content for display tothe user, and an integrated image source for introducing the content tothe optical assembly. The displayed content may comprise an interactivekeyboard control element (e.g. a virtual keyboard, calculator, touchscreen, and the like), and where the keyboard control element isassociated with an input path analyzer, a word matching search facility,and a keyboard input interface. The user may input text by sliding apointing device (e.g. a finger, a stylus, and the like) across characterkeys of the keyboard input interface in a sliding motion through anapproximate sequence of a word the user would like to input as text,wherein the input path analyzer determines the characters contacted inthe input path, the word matching facility finds a best word match tothe sequence of characters contacted and inputs the best word match asinput text. In embodiments, the reference displayed content may besomething other than a keyboard, such as a sketch pad for freehand text,or other interface references like a 4-way joystick pad for controllinga game or real robots and aircraft, and the like. Another example may bea virtual drum kit, such as with colored pads the user “taps” to make asound. The eyepiece's ability to interpret patterns of motion across asurface may allow for projecting reference content in order to give theuser something to point at and provide them with visual and/or audiofeedback. In embodiments, the ‘motion’ detected by the eyepiece may bethe motion of the user's eye as they look at the surface. For example,the eyepiece may have facilities for tracking the eye movement of theuser, and by having both the content display locations of a projectedvirtual keyboard and the gazing direction of the user's eye, theeyepiece may be able to detect the line-of-sight motion of the user'seye across the keyboard, and then interpret the motions as words asdescribed herein.

In embodiments, the eyepiece may provide the capability to command theeyepiece via hand gesture ‘air lettering’, such as the wearer usingtheir finger to air swipe out a letter, word, and the like in view of anembedded eyepiece camera, where the eyepiece interprets the fingermotion as letters, words, symbols for commanding, signatures, writing,emailing, texting, and the like. For instance, the wearer may use thistechnique to sign a document utilizing an ‘air signature’. The wearermay use this technique to compose text, such as in an email, text,document, and the like. The wearer eyepiece may recognize a symbol madethrough the hand motion as a control command. In embodiments, the airlettering may be implemented through hand gesture recognition asinterpreted by images captured through an eyepiece camera, or throughother input control devices, such as via an inertial measurement unit(IMU) mounted in a device on the user's finger, hand, and the like, asdescribed herein.

In embodiments, eyepiece facilities may provide for presenting displayedcontent corresponding to an identified marker indicative of theintention to display the content. That is, the eyepiece may be commandedto display certain content based upon sensing a predetermined externalvisual cue. The visual cue may be an image, an icon, a picture, facerecognition, a hand configuration, a body configuration, and the like.The displayed content may be an interface device that is brought up foruse, a navigation aid to help the user find a location once they get tosome travel location, an advertisement when the eyepiece views a targetimage, an informational profile, and the like. In embodiments, visualmarker cues and their associated content for display may be stored inmemory on the eyepiece, in an external computer storage facility andimported as needed (such as by geographic location, proximity to atrigger target, command by the user, and the like), generated by athird-party, and the like. In embodiments, the user may wear theinteractive head-mounted eyepiece, where the eyepiece includes anoptical assembly through which the user views a surrounding environmentand displayed content. The optical assembly may include a correctiveelement that corrects the user's view of the surrounding environment, anintegrated processor for handling content for display to the user, andan integrated image source for introducing the content to the opticalassembly. An integrated camera facility may be provided that images anexternal visual cue, wherein the integrated processor identifies andinterprets the external visual cue as a command to display contentassociated with the visual cue. Referring to FIG. 38, in embodiments thevisual cue 3812 may be included in a sign 3814 in the surroundingenvironment, where the projected content is associated with anadvertisement. The sign may be a billboard, and the advertisement for apersonalized advertisement based on a preferences profile of the user.The visual cue 3802,3808 may be a hand gesture, and the projectedcontent a projected virtual keyboard 3804, 3810. For instance, the handgesture may be a thumb and index finger gesture 3802 from a first userhand, and the virtual keyboard 3804 projected on the palm of the firstuser hand, and where the user is able to type on the virtual keyboardwith a second user hand. The hand gesture 3808 may be a thumb and indexfinger gesture combination of both user hands, and the virtual keyboard3810 projected between the user hands as configured in the hand gesture,where the user is able to type on the virtual keyboard using the thumbsof the user's hands. Visual cues may provide the wearer of the eyepiecewith an automated resource for associating a predetermined externalvisual cue with a desired outcome in the way of projected content, thusfreeing the wearer from searching for the cues themselves.

In embodiments, the eyepiece may include a visual recognition languagetranslation facility for providing translations for visually presentedcontent, such as for road signs, menus, billboards, store signs, books,magazines, and the like. The visual recognition language translationfacility may utilize optical character recognition to identify lettersfrom the content, match the strings of letters to words and phrasesthrough a database of translations. This capability may be completelycontained within the eyepiece, such as in an offline mode, or at leastin part in an external computing facility, such as on an externalserver. For instance, a user may be in a foreign country, where thesigns, menus, and the like are not understood by the wearer of theeyepiece, but for which the eyepiece is able to provide translations.These translations may appear as an annotation to the user, replace theforeign language words (such as on the sign) with the translation,provided through an audio translation to the user, and the like. In thisway, the wearer won't have to take the effort to look up wordtranslations, but rather they would be provided automatically. In anexample, a user of the eyepiece may be Italian, and coming to the UnitedStates they have the need to interpret the large number of road signs inorder to drive around safely. Referring to FIG. 38A, the Italian user ofthe eyepiece is viewing a U.S. stop sign 3802A. In this instance, theeyepiece may identify the letters on the sign, translate the word ‘stop’in the Italian for stop, ‘arresto’, and make the stop sign 3804A appearto read the word ‘arresto’ rather than ‘stop’. In embodiments, theeyepiece may also provide simple translation messages to the wearer,provide audio translations, provide a translation dictionary to thewearer, and the like.

In one example, the eyepiece may be used in an adaptive environment,such as for blind users. In embodiments, the results of face recognitionor object identification may be processed to obtain an audible resultand can be presented as audio to a wearer of the glasses throughassociated earbuds/headphones. In other embodiments, the results of facerecognition or object identification may be translated into hapticvibrations in the glasses or an associated controller. In an example, ifsomeone stands in front of a user of the adaptive glasses, a camera mayimage the person and transmit the image to the integrated processor forprocessing by face recognition software or to face recognition softwareoperating on a server or in the cloud. The results of the facerecognition may be presented as written text in the display of theglasses for certain individuals, but for blind or poor vision users, theresult may be processed to obtain audio. In other examples, objectrecognition may determine the user is approaching a curb, doorway, orother object and the glasses or controller would audibly or hapticallywarn the user. For poor vision users, the text on the display could bemagnified or the contrast could be increased.

In embodiments, a GPS sensor may be used to determine a location of theuser wearing the adaptive display. The GPS sensor may be accessed by anavigation application to audibly announce various points of interest tothe user as they are approached or reached. In embodiments, the user maybe audibly guided to an endpoint by the navigation application.

The eyepiece may be useful for various applications and markets. Itshould be understood that the control mechanisms described herein may beused to control the functions of the applications described herein. Theeyepiece may run a single application at a time or multiple applicationsmay run at a time. Switching between applications may be done with thecontrol mechanisms described herein. The eyepiece may be used inmilitary applications, gaming, image recognition applications, toview/order e-books, GPS Navigation (Position, Direction, Speed and ETA),Mobile TV, athletics (view pacing, ranking, and competition times;receive coaching), telemedicine, industrial inspection, aviation,shopping, inventory management tracking, firefighting (enabled byVIS/NIRSWIR sensor that sees through fog, haze, dark),outdoor/adventure, custom advertising, and the like. In an embodiment,the eyepiece may be used with e-mail, such as GMAIL in FIG. 7, theInternet, web browsing, viewing sports scores, video chat, and the like.In an embodiment, the eyepiece may be used for educational/trainingpurposes, such as by displaying step by step guides, such as hands-free,wireless maintenance and repair instructions. For example, a videomanual and/or instructions may be displayed in the field of view. In anembodiment, the eyepiece may be used in Fashion, Health, and Beauty. Forexample, potential outfits, hairstyles, or makeup may be projected ontoa mirror image of a user. In an embodiment, the eyepiece may be used inBusiness Intelligence, Meetings, and Conferences. For example, a user'sname tag can be scanned, their face run through a facial recognitionsystem, or their spoken name searched in database to obtain biographicalinformation. Scanned name tags, faces, and conversations may be recordedfor subsequent viewing or filing.

In an embodiment, a “Mode” may be entered by the eyepiece. In the mode,certain applications may be available. For example, a consumer versionof the eyepiece may have a Tourist Mode, Educational Mode, InternetMode, TV Mode, Gaming Mode, Exercise Mode, Stylist Mode, PersonalAssistant Mode, and the like.

A user of the augmented reality glasses may wish to participate in videocalling or video conferencing while wearing the glasses. Many computers,both desktop and laptop have integrated cameras to facilitate usingvideo calling and conferencing. Typically, software applications areused to integrate use of the camera with calling or conferencingfeatures. With the augmented reality glasses providing much of thefunctionality of laptops and other computing devices, many users maywish to utilize video calling and video conferencing while on the movewearing the augmented reality glasses.

In an embodiment, a video calling or video conferencing application maywork with a WiFi connection, or may be part of a 3G or 4G callingnetwork associated with a user's cell phone. The camera for videocalling or conferencing is placed on a device controller, such as awatch or other separate electronic computing device. Placing the videocalling or conferencing camera on the augmented reality glasses is notfeasible, as such placement would provide the user with a view only ofthemselves, and would not display the other participants in theconference or call. However, the user may choose to use theforward-facing camera to display their surroundings or anotherindividual in the video call.

FIG. 32 depicts a typical camera 3200 for use in video calling orconferencing. Such cameras are typically small and could be mounted on awatch 3202, as shown in FIG. 32, cell phone or other portable computingdevice, including a laptop computer. Video calling works by connectingthe device controller with the cell phone or other communicationsdevice. The devices utilize software compatible with the operatingsystem of the glasses and the communications device or computing device.In an embodiment, the screen of the augmented reality glasses maydisplay a list of options for making the call and the user may gestureusing a pointing control device or use any other control techniquedescribed herein to select the video calling option on the screen of theaugmented reality glasses.

FIG. 33 illustrates an embodiment 3300 of a block diagram of avideo-calling camera. The camera incorporates a lens 3302, a CCD/CMOSsensor 3304, analog to digital converters for video signals, 3306, andaudio signals, 3314. Microphone 3312 collects audio input. Both analogto digital converters 3306 and 3314 send their output signals to asignal enhancement module 3308. The signal enhancement module 3308forwards the enhanced signal, which is a composite of both video andaudio signals to interface 3310. Interface 3310 is connected to an IEEE1394 standard bus interface, along with a control module 3316.

In operation, the video call camera depends on the signal capture whichtransforms the incident light, as well as incident sound into electrons.For light this process is performed by CCD or CMOS chip 3304. Themicrophone transforms sound into electrical impulses.

The first step in the process of generating an image for a video call isto digitize the image. The CCD or CMOS chip 3304 dissects the image andconverts it into pixels. If a pixel has collected many photons, thevoltage will be high. If the pixel has collected few photons, thevoltage will be low. This voltage is an analog value. During the secondstep of digitization, the voltage is transformed into a digital value bythe analog to digital converter 3306, which handles image processing. Atthis point, a raw digital image is available.

Audio captured by the microphone 3312 is also transformed into avoltage. This voltage is sent to the analog to digital converter 3314where the analog values are transformed into digital values.

The next step is to enhance the signal so that it may be sent to viewersof the video call or conference. Signal enhancement includes creatingcolor in the image using a color filter, located in front of the CCD orCMOS chip 3304. This filter is red, green, or blue and changes its colorfrom pixel to pixel, and in an embodiment, may be a color filter array,or Bayer filter. These raw digital images are then enhanced by thefilter to meet aesthetic requirements. Audio data may also be enhancedfor a better calling experience.

In the final step before transmission, the image and audio data arecompressed and output as a digital video stream, in an embodiment usinga digital video camera. If a photo camera is used, single images may beoutput, and in a further embodiment, voice comments may be appended tothe files. The enhancement of the raw digital data takes place away fromthe camera, and in an embodiment may occur in the device controller orcomputing device that the augmented reality glasses communicate withduring a video call or conference.

Further embodiments may provide for portable cameras for use inindustry, medicine, astronomy, microscopy, and other fields requiringspecialized camera use. These cameras often forgo signal enhancement andoutput the raw digital image. These cameras may be mounted on otherelectronic devices or the user's hand for ease of use.

The camera interfaces to the augmented reality glasses and the devicecontroller or computing device using an IEEE 1394 interface bus. Thisinterface bus transmits time critical data, such as a video and datawhose integrity is critically important, including parameters or filesto manipulate data or transfer images.

In addition to the interface bus, protocols define the behavior of thedevices associated with the video call or conference. The camera for usewith the augmented reality glasses, may, in embodiments, employ one ofthe following protocols: AV/C, DCAM, or SBP-2.

AV/C is a protocol for Audio Video Control and defines the behavior ofdigital video devices, including video cameras and video recorders.

DCAM refers to the 1394 based Digital Camera Specification and definesthe behavior of cameras that output uncompressed image data withoutaudio.

SBP-2 refers to Serial Bus Protocol and defines the behavior of massstorage devices, such as hard drives or disks.

Devices that use the same protocol are able to communicate with eachother. Thus, for video calling using the augmented reality glasses, thesame protocol may be used by the video camera on the device controllerand the augmented reality glasses. Because the augmented realityglasses, device controller, and camera use the same protocol, data maybe exchanged among these devices. Files that may be transferred amongdevices include: image and audio files, image and audio data flows,parameters to control the camera, and the like.

In an embodiment, a user desiring to initiate a video call may select avideo call option from a screen presented when the call process isinitiated. The user selects by making a gesture using a pointing device,or gesture to signal the selection of the video call option. The userthen positions the camera located on the device controller, wristwatch,or other separable electronic device so that the user's image iscaptured by the camera. The image is processed through the processdescribed above and is then streamed to the augmented reality glassesand the other participants for display to the users.

In embodiments, the camera may be mounted on a cell phone, personaldigital assistant, wristwatch, pendant, or other small portable devicecapable of being carried, worn, or mounted. The images or video capturedby the camera may be streamed to the eyepiece. For example, when acamera is mounted on a rifle, a wearer may be able to image targets notin the line of sight and wirelessly receive imagery as a stream ofdisplayed content to the eyepiece.

In embodiments, the present disclosure may provide the wearer withGPS-based content reception, as in FIG. 6. As noted, augmented realityglasses of the present disclosure may include memory, a globalpositioning system, a compass or other orienting device, and a camera.GPS-based computer programs available to the wearer may include a numberof applications typically available from the Apple Inc. App Store foriPhone use. Similar versions of these programs are available for otherbrands of smart phone and may be applied to embodiments of the presentdisclosure. These programs include, for example, SREngine (scenerecognition engine), NearestTube, TAT Augmented ID, Yelp, Layar, andTwittARound, as well as other more specialized applications, such asRealSki.

SREngine is a scene recognition engine that is able to identify objectsviewed by the user's camera. It is a software engine able to recognizestatic scenes, such as scenes of architecture, structures, pictures,objects, rooms, and the like. It is then able to automatically apply avirtual “label” to the structures or objects according to what itrecognizes. For example, the program may be called up by a user of thepresent disclosure when viewing a street scene, such as FIG. 6. Using acamera of the augmented reality glasses, the engine will recognize theFontaines de la Concorde in Paris. The program will then summon avirtual label, shown in FIG. 6 as part of a virtual image 618 projectedonto the lens 602. The label may be text only, as seen at the bottom ofthe image 618. Other labels applicable to this scene may include“fountain,” “museum,” “hotel,” or the name of the columned building inthe rear. Other programs of this type may include the Wikitude AR TravelGuide, Yelp and many others.

NearestTube, for example, uses the same technology to direct a user tothe closest subway station in London, and other programs may perform thesame function, or similar, in other cities. Layar is another applicationthat uses the camera, a compass or direction, and GPS data to identify auser's location and field of view. With this information, an overlay orlabel may appear virtually to help orient and guide the user. Yelp andMonocle perform similar functions, but their databases are somewhat morespecialized, helping to direct users in a similar manner to restaurantsor to other service providers.

The user may control the glasses, and call up these functions, using anyof the controls described in this patent. For example, the glasses maybe equipped with a microphone to pick up voice commands from a user andprocess them using software contained with a memory of the glasses. Theuser may then respond to prompts from small speakers or earbuds alsocontained within the glasses frame. The glasses may also be equippedwith a tiny track pad, similar to those found on smartphones. Thetrackpad may allow a user to move a pointer or indicator on the virtualscreen within the AR glasses, similar to a touch screen. When the userreaches a desired point on the screen, the user depresses the track padto indicate his or her selection. Thus, a user may call up a program,e.g., a travel guide, and then find his or her way through severalmenus, perhaps selecting a country, a city and then a category. Thecategory selections may include, for example, hotels, shopping, museums,restaurants, and so forth. The user makes his or her selections and isthen guided by the AR program. In one embodiment, the glasses alsoinclude a GPS locator, and the present country and city provides defaultlocations that may be overridden.

In an embodiment, the eyepiece's object recognition software may processthe images being received by the eyepiece's forward facing camera inorder to determine what is in the field of view. In other embodiments,the GPS coordinates of the location as determined by the eyepiece's GPSmay be enough to determine what is in the field of view. In otherembodiments, an RFID or other beacon in the environment may bebroadcasting a location. Any one or combination of the above may be usedby the eyepiece to identify the location and the identity of what is inthe field of view.

When an object is recognized, the resolution for imaging that object maybe increased or images or video may be captured at low compression.Additionally, the resolution for other objects in the user's view may bedecreased, or captured at a higher compression rate in order to decreasethe needed bandwidth.

Once determined, content related to points of interest in the field ofview may be overlaid on the real world image, such as social networkingcontent, interactive tours, local information, and the like. Informationand content related to movies, local information, weather, restaurants,restaurant availability, local events, local taxis, music, and the likemay be accessed by the eyepiece and projected on to the lens of theeyepiece for the user to view and interact with. For example, as theuser looks at the Eiffel Tower, the forward facing camera may take animage and send it for processing to the eyepiece's associated processor.Object recognition software may determine that the structure in thewearer's field of view is the Eiffel Tower. Alternatively, the GPScoordinates determined by the eyepiece's GPS may be searched in adatabase to determine that the coordinates match those of the EiffelTower. In any event, content may then be searched relating to the EiffelTower visitor's information, restaurants in the vicinity and in theTower itself, local weather, local Metro information, local hotelinformation, other nearby tourist spots, and the like. Interacting withthe content may be enabled by the control mechanisms described herein.In an embodiment, GPS-based content reception may be enabled when aTourist Mode of the eyepiece is entered.

In an embodiment, the eyepiece may be used to view streaming video. Forexample, videos may be identified via search by GPS location, search byobject recognition of an object in the field of view, a voice search, aholographic keyboard search, and the like. Continuing with the exampleof the Eiffel Tower, a video database may be searched via the GPScoordinates of the Tower or by the term ‘Eiffel Tower’ once it has beendetermined that is the structure in the field of view. Search resultsmay include geo-tagged videos or videos associated with the EiffelTower. The videos may be scrolled or flipped through using the controltechniques described herein. Videos of interest may be played using thecontrol techniques described herein. The video may be laid over the realworld scene or may be displayed on the lens out of the field of view. Inan embodiment, the eyepiece may be darkened via the mechanisms describedherein to enable higher contrast viewing. In another example, theeyepiece may be able to utilize a camera and network connectivity, suchas described herein, to provide the wearer with streaming videoconferencing capabilities.

As noted, the user of augmented reality may receive content from anabundance of sources. A visitor or tourist may desire to limit thechoices to local businesses or institutions; on the other hand,businesses seeking out visitors or tourists may wish to limit theiroffers or solicitations to persons who are in their area or location butwho are visiting rather than local residents. Thus, in one embodiment,the visitor or tourist may limit his or her search only to localbusinesses, say those within certain geographic limits. These limits maybe set via GPS criteria or by manually indicating a geographicrestriction. For example, a person may require that sources of streamingcontent or ads be limited to those within a certain radius (a set numberor km or miles) of the person. Alternatively, the criteria may requirethat the sources are limited to those within a certain city or province.These limits may be set by the augmented reality user just as a user ofa computer at a home or office would limit his or her searches using akeyboard or a mouse; the entries for augmented reality users are simplymade by voice, by hand motion, or other ways described elsewhere in theportions of this disclosure discussing controls.

In addition, the available content chosen by a user may be restricted orlimited by the type of provider. For example, a user may restrictchoices to those with a website operated by a government institution(.gov) or by a non-profit institution or organization (.org). In thisway, a tourist or visitor who may be more interested in visitinggovernment offices, museums, historical sites and the like, may find hisor her choices less cluttered. The person may be more easily able tomake decisions when the available choices have been pared down to a morereasonable number. The ability to quickly cut down the available choicesis desirable in more urban areas, such as Paris or Washington, D.C.,where there are many choices.

The user controls the glasses in any of the manners or modes describedelsewhere in this patent. For example, the user may call up a desiredprogram or application by voice or by indicating a choice on the virtualscreen of the augmented reality glasses. The augmented glasses mayrespond to a track pad mounted on the frame of the glasses, as describedabove. Alternatively, the glasses may be responsive to one or moremotion or position sensors mounted on the frame. The signals from thesensors are then sent to a microprocessor or microcontroller within theglasses, the glasses also providing any needed signal transducing orprocessing. Once the program of choice has begun, the user makesselections and enters a response by any of the methods discussed herein,such as signaling “yes” or “no” with a head movement, a hand gesture, atrackpad depression, or a voice command.

At the same time, content providers, that is, advertisers, may also wishto restrict their offerings to persons who are within a certaingeographic area, e.g., their city limits. At the same time, anadvertiser, perhaps a museum, may not wish to offer content to localpersons, but may wish to reach visitors or out-of-towners. In anotherexample, advertisements may not be presented when the user is home butmay be presented when the user is traveling or away from home. Theaugmented reality devices discussed herein are desirably equipped withboth GPS capability and telecommunications capability and an integratedprocessor for implementing geographic-based rules for advertisementpresentation. It will be a simple matter for the museum to providestreaming content within a limited area by limiting its broadcast power.The museum, however, may provide the content through the Internet andits content may be available world-wide. In this instance, a user mayreceive content through an augmented reality device advising that themuseum is open today and is available for touring.

The user may respond to the content by the augmented reality equivalentof clicking on a link for the museum. The augmented reality equivalentmay be a voice indication, a hand or eye movement, or other sensoryindication of the user's choice, or by using an associated body-mountedcontroller. The museum then receives a cookie indicating the identity ofthe user or at least the user's internet service provider (ISP). If thecookie indicates or suggests an internet service provider other thanlocal providers, the museum server may then respond with advertisementsor offers tailored to visitors. The cookie may also include anindication of a telecommunications link, e.g., a telephone number. Ifthe telephone number is not a local number, this is an additional cluethat the person responding is a visitor. The museum or other institutionmay then follow up with the content desired or suggested by itsmarketing department.

Another application of the augmented reality eyepiece takes advantage ofa user's ability to control the eyepiece and its tools with a minimumuse of the user's hands, using instead voice commands, gestures ormotions. As noted above, a user may call upon the augmented realityeyepiece to retrieve information. This information may already be storedin a memory of the eyepiece, but may instead be located remotely, suchas a database accessible over the Internet or perhaps via an intranetwhich is accessible only to employees of a particular company ororganization. The eyepiece may thus be compared to a computer or to adisplay screen which can be viewed and heard at an extremely close rangeand generally controlled with a minimal use of one's hands.

Applications may thus include providing information on-the-spot to amechanic or electronics technician. The technician can don the glasseswhen seeking information about a particular structure or problemencountered, for example, when repairing an engine or a power supply.Using voice commands, he or she may then access the database and searchwithin the database for particular information, such as manuals or otherrepair and maintenance documents. The desired information may thus bepromptly accessed and applied with a minimum of effort, allowing thetechnician to more quickly perform the needed repair or maintenance andto return the equipment to service. For mission-critical equipment, suchtime savings may also save lives, in addition to saving repair ormaintenance costs.

The information imparted may include repair manuals and the like, butmay also include a full range of audio-visual information, i.e., theeyepiece screen may display to the technician or mechanic a video of howto perform a particular task at the same time the person is attemptingto perform the task. The augmented reality device also includestelecommunications capabilities, so the technician also has the abilityto call on others to assist if there is some complication or unexpecteddifficulty with the task. This educational aspect of the presentdisclosure is not limited to maintenance and repair, but may be appliedto any educational endeavor, such as secondary or post-secondaryclasses, continuing education courses or topics, seminars, and the like.

In an embodiment, a Wi-Fi enabled eyepiece may run a location-basedapplication for geo-location of opted-in users. Users may opt-in bylogging into the application on their phone and enabling broadcast oftheir location, or by enabling geo-location on their own eyepiece. As awearer of the eyepiece scans people, and thus their opted-in device, theapplication may identify opted-in users and send an instruction to theprojector to project an augmented reality indicator on an opted-in userin the user's field of view. For example, green rings may be placedaround people who have opted-in to have their location seen. In anotherexample, yellow rings may indicate people who have opted-in but don'tmeet some criteria, such as they do not have a FACEBOOK account, or thatthere are no mutual friends if they do have a FACEBOOK account.

Some social networking, career networking, and dating applications maywork in concert with the location-based application. Software residenton the eyepiece may coordinate data from the networking and dating sitesand the location-based application. For example, TwittARound is one suchprogram which makes use of a mounted camera to detect and labellocation-stamped tweets from other tweeters nearby. This will enable aperson using the present disclosure to locate other nearby Twitterusers. Alternatively, users may have to set their devices to coordinateinformation from various networking and dating sites. For example, thewearer of the eyepiece may want to see all E-HARMONY users who arebroadcasting their location. If an opted-in user is identified by theeyepiece, an augmented reality indicator may be laid over the opted-inuser. The indicator may take on a different appearance if the user hassomething in common with the wearer, many things in common with theuser, and the like. For example, and referring to FIG. 16, two peopleare being viewed by the wearer. Both of the people are identified asE-HARMONY users by the rings placed around them. However, the womanshown with solid rings has more than one item in common with the wearerwhile the woman shown with dotted rings has no items in common with thewearer. Any available profile information may get accessed and displayedto the user.

In an embodiment, when the wearer directs the eyepiece in the directionof a user who has a networking account, such as FACEBOOK, TWITTER,BLIPPY, LINKEDIN, GOOGLE, WIKIPEDIA, and the like, the user's recentposts or profile information may be displayed to the wearer. Forexample, recent status updates, “tweets”, “blips”, and the like may getdisplayed, as mentioned above for TwittARound. In an embodiment, whenthe wearer points the eyepiece in a target user's direction, they mayindicate interest in the user if the eyepiece is pointed for a durationof time and/or a gesture, head, eye, or audio control is activated. Thetarget user may receive an indication of interest on their phone or intheir glasses. If the target user had marked the wearer as interestingbut was waiting on the wearer to show interest first, an indication mayimmediately pop up in the eyepiece of the target user's interest. Acontrol mechanism may be used to capture an image and store the targetuser's information on associated non-volatile memory or in an onlineaccount.

In other applications for social networking, a facial recognitionprogram, such as TAT Augmented ID, from TAT—The Astonishing Tribe,Malmö, Sweden, may be used. Such a program may be used to identify aperson by his or her facial characteristics. This software uses facialrecognition software to identify a person. Using other applications,such as photo identifying software from Flickr, one can then identifythe particular nearby person, and one can then download information fromsocial networking sites with information about the person. Thisinformation may include the person's name and the profile the person hasmade available on sites such as Facebook, Twitter, and the like. Thisapplication may be used to refresh a user's memory of a person or toidentify a nearby person, as well as to gather information about theperson.

In other applications for social networking, the wearer may be able toutilize location-based facilities of the eyepiece to leave notes,comments, reviews, and the like, at locations, in association withpeople, places, products, and the like. For example, a person may beable to post a comment on a place they visited, where the posting maythen be made available to others through the social network. In anotherexample, a person may be able to post that comment at the location ofthe place such that the comment is available when another person comesto that location. In this way, a wearer may be able to access commentsleft by others when they come to the location. For instance, a wearermay come to the entrance to a restaurant, and be able to access reviewsfor the restaurant, such as sorted by some criteria (e.g. most recentreview, age of reviewer, and the like).

A user may initiate the desired program by voice, by selecting a choicefrom a virtual touchscreen, as described above, by using a trackpad toselect and choose the desired program, or by any of the controltechniques described herein. Menu selections may then be made in asimilar or complementary manner. Sensors or input devices mounted inconvenient locations on the user's body may also be used, e.g., sensorsand a track pad mounted on a wrist pad, on a glove, or even a discreetdevice, perhaps of the size of a smart phone or a personal digitalassistant.

Applications of the present disclosure may provide the wearer withInternet access, such as for browsing, searching, shopping,entertainment, and the like, such as through a wireless communicationsinterface to the eyepiece. For instance, a wearer may initiate a websearch with a control gesture, such as through a control facility wornon some portion of the wearer's body (e.g. on the hand, the head, thefoot), on some component being used by the wearer (e.g. a personalcomputer, a smart phone, a music player), on a piece of furniture nearthe wearer (e.g. a chair, a desk, a table, a lamp), and the like, wherethe image of the web search is projected for viewing by the wearerthrough the eyepiece. The wearer may then view the search through theeyepiece and control web interaction though the control facility.

In an example, a user may be wearing an embodiment configured as a pairof glasses, with the projected image of an Internet web browser providedthrough the glasses while retaining the ability to simultaneously viewat least portions of the surrounding real environment. In this instance,the user may be wearing a motion sensitive control facility on theirhand, where the control facility may transmit relative motion of theuser's hand to the eyepiece as control motions for web control, such assimilar to that of a mouse in a conventional personal computerconfiguration. It is understood that the user would be enabled toperform web actions in a similar fashion to that of a conventionalpersonal computer configuration. In this case, the image of the websearch is provided through the eyepiece while control for selection ofactions to carry out the search is provided though motions of the hand.For instance, the overall motion of the hand may move a cursor withinthe projected image of the web search, the flick of the finger(s) mayprovide a selection action, and so forth. In this way, the wearer may beenabled to perform the desired web search, or any other Internetbrowser-enabled function, through an embodiment connected to theInternet. In one example, a user may have downloaded computer programsYelp or Monocle, available from the App Store, or a similar product,such as NRU (“near you”), an application from Zagat to locate nearbyrestaurants or other stores, Google Earth, Wikipedia, or the like. Theperson may initiate a search, for example, for restaurants, or otherproviders of goods or services, such as hotels, repairmen, and the like,or information. When the desired information is found, locations aredisplayed or a distance and direction to a desired location isdisplayed. The display may take the form of a virtual label co-locatedwith the real world object in the user's view.

Other applications from Layar (Amsterdam, the Netherlands) include avariety of “layers” tailored for specific information desired by a user.A layer may include restaurant information, information about a specificcompany, real estate listings, gas stations, and so forth. Using theinformation provided in a software application, such as a mobileapplication and a user's global positioning system (GPS), informationmay be presented on a screen of the glasses with tags having the desiredinformation. Using the haptic controls or other control discussedelsewhere in this disclosure, a user may pivot or otherwise rotate hisor her body and view buildings tagged with virtual tags containinginformation. If the user seeks restaurants, the screen will displayrestaurant information, such as name and location. If a user seeks aparticular address, virtual tags will appear on buildings in the fieldof view of the wearer. The user may then make selections or choices byvoice, by trackpad, by virtual touch screen, and so forth.

Applications of the present disclosure may provide a way foradvertisements to be delivered to the wearer. For example,advertisements may be displayed to the viewer through the eyepiece asthe viewer is going about his or her day, while browsing the Internet,conducting a web search, walking though a store, and the like. Forinstance, the user may be performing a web search, and through the websearch the user is targeted with an advertisement. In this example, theadvertisement may be projected in the same space as the projected websearch, floating off to the side, above, or below the view angle of thewearer. In another example, advertisements may be triggered for deliveryto the eyepiece when some advertising providing facility, perhaps one inproximity to the wearer, senses the presence of the eyepiece (e.g.through a wireless connection, RFID, and the like), and directs theadvertisement to the eyepiece. In embodiments, the eyepiece may be usedfor tracking of advertisement interactions, such as the user seeing orinteracting with a billboard, a promotion, an advertisement, and thelike. For instance, user's behavior with respect to advertisements maybe tracked, such as to provide benefits, rewards, and the like to theuser. In an example, the user may be paid five dollars in virtual cashwhenever they see a billboard. The eyepiece may provide impressiontracking, such as based on seeing branded images (e.g. based on time,geography), and the like. As a result, offers may be targeted based onthe location and the event related to the eyepiece, such as what theuser saw, heard, interacted with, and the like. In embodiments, adtargeting may be based on historical behavior, such as based on what theuser has interacted with in the past, patterns of interactions, and thelike.

For example, the wearer may be window-shopping in Manhattan, wherestores are equipped with such advertising providing facilities. As thewearer walks by the stores, the advertising providing facilities maytrigger the delivery of an advertisement to the wearer based on a knownlocation of the user determined by an integrated location sensor of theeyepiece, such as a GPS. In an embodiment, the location of the user maybe further refined via other integrated sensors, such as a magnetometerto enable hyperlocal augmented reality advertising. For example, a useron a ground floor of a mall may receive certain advertisements if themagnetometer and GPS readings place the user in front of a particularstore. When the user goes up one flight in the mall, the GPS locationmay remain the same, but the magnetometer reading may indicate a changein elevation of the user and a new placement of the user in front of adifferent store. In embodiments, one may store personal profileinformation such that the advertising providing facility is able tobetter match advertisements to the needs of the wearer, the wearer mayprovide preferences for advertisements, the wearer may block at leastsome of the advertisements, and the like. The wearer may also be able topass advertisements, and associated discounts, on to friends. The wearermay communicate them directly to friends that are in close proximity andenabled with their own eyepiece; they may also communicate them througha wireless Internet connection, such as to a social network of friends,though email, SMS; and the like. The wearer may be connected tofacilities and/or infrastructure that enables the communication ofadvertisements from a sponsor to the wearer; feedback from the wearer toan advertisement facility, the sponsor of the advertisement, and thelike; to other users, such as friends and family, or someone inproximity to the wearer; to a store, such as locally on the eyepiece orin a remote site, such as on the Internet or on a user's home computer;and the like. These interconnectivity facilities may include integratedfacilities to the eyepiece to provide the user's location and gazedirection, such as through the use of GPS, 3-axis sensors, magnetometer,gyros, accelerometers, and the like, for determining direction, speed,attitude (e.g. gaze direction) of the wearer. Interconnectivityfacilities may provide telecommunications facilities, such as cellularlink, a WiFi/MiFi bridge, and the like. For instance, the wearer may beable to communicate through an available WiFi link, through anintegrated MiFi (or any other personal or group cellular link) to thecellular system, and the like. There may be facilities for the wearer tostore advertisements for a later use. There may be facilities integratedwith the wearer's eyepiece or located in local computer facilities thatenable caching of advertisements, such as within a local area, where thecached advertisements may enable the delivery of the advertisements asthe wearer nears the location associated with the advertisement. Forexample, local advertisements may be stored on a server that containsgeo-located local advertisements and specials, and these advertisementsmay be delivered to the wearer individually as the wearer approaches aparticular location, or a set of advertisements may be delivered to thewearer in bulk when the wearer enters a geographic area that isassociated with the advertisements so that the advertisements areavailable when the user nears a particular location. The geographiclocation may be a city, a part of the city, a number of blocks, a singleblock, a street, a portion of the street, sidewalk, and the like,representing regional, local, hyper-local areas. Note that the precedingdiscussion uses the term advertisement, but one skilled in the art willappreciate that this can also mean an announcement, a broadcast, acircular, a commercial, a sponsored communication, an endorsement, anotice, a promotion, a bulletin, a message, and the like.

FIGS. 18-20A depict ways to deliver custom messages to persons within ashort distance of an establishment that wishes to send a message, suchas a retail store. Referring to FIG. 18 now, embodiments may provide fora way to view custom billboards, such as when the wearer of the eyepieceis walking or driving, by applications as mentioned above for searchingfor providers of goods and services. As depicted in FIG. 18, thebillboard 1800 shows an exemplary augmented reality-based advertisementdisplayed by a seller or a service provider. The exemplaryadvertisement, as depicted, may relate to an offer on drinks by a bar.For example, two drinks may be provided for the cost of just one drink.With such augmented reality-based advertisements and offers, thewearer's attention may be easily directed towards the billboards. Thebillboards may also provide details about location of the bar such asstreet address, floor number, phone number, and the like. In accordancewith other embodiments, several devices other than eyepiece may beutilized to view the billboards. These devices may include withoutlimitations smart phones, IPHONEs, IPADs, car windshields, user glasses,helmets, wristwatches, headphones, vehicle mounts, and the like. Inaccordance with an embodiment, a user (wearer in case the augmentedreality technology is embedded in the eyepiece) may automaticallyreceive offers or view a scene of the billboards as and when the userpasses or drives by the road. In accordance with another embodiment, theuser may receive offers or view the scene of the billboards based on hisrequest.

FIG. 19 illustrates two exemplary roadside billboards 1900 containingoffers and advertisements from sellers or service providers that may beviewed in the augmented reality manner. The augmented advertisement mayprovide a live and near-to-reality perception to the user or the wearer.

As illustrated in FIG. 20, the augmented reality enabled device such asthe camera lens provided in the eyepiece may be utilized to receiveand/or view graffiti 2000, slogans, drawings, and the like, that may bedisplayed on the roadside or on top, side, front of the buildings andshops. The roadside billboards and the graffiti may have a visual (e.g.a code, a shape) or wireless indicator that may link the advertisement,or advertisement database, to the billboard. When the wearer nears andviews the billboard, a projection of the billboard advertisement maythen be provided to the wearer. In embodiments, one may also storepersonal profile information such that the advertisements may bettermatch the needs of the wearer, the wearer may provide preferences foradvertisements, the wearer may block at least some of theadvertisements, and the like. In embodiments, the eyepiece may havebrightness and contrast control over the eyepiece projected area of thebillboard so as to improve readability for the advertisement, such as ina bright outside environment.

In other embodiments, users may post information or messages on aparticular location, based on its GPS location or other indicator oflocation, such as a magnetometer reading. The intended viewer is able tosee the message when the viewer is within a certain distance of thelocation, as explained with FIG. 20A. In a first step 2001 of the methodFIG. 20A, a user decides the location where the message is to bereceived by persons to whom the message is sent. The message is thenposted 2003, to be sent to the appropriate person or persons when therecipient is close to the intended “viewing area.” Location of thewearers of the augmented reality eyepiece is continuously updated 2005by the GPS system which forms a part of the eyepiece. When the GPSsystem determines that the wearer is within a certain distance of thedesired viewing area, e.g., 10 meters, the message is then sent 2007 tothe viewer. In one embodiment, the message then appears as e-mail or atext message to the recipient, or if the recipient is wearing aneyepiece, the message may appear in the eyepiece. Because the message issent to the person based on the person's location, in one sense, themessage may be displayed as “graffiti” on a building or feature at ornear the specified location. Specific settings may be used to determineif all passersby to the “viewing area” can see the message or if only aspecific person or group of people or devices with specific identifiers.For example, a soldier clearing a village may virtually mark a house ascleared by associating a message or identifier with the house, such as abig X marking the location of the house. The soldier may indicate thatonly other American soldiers may be able to receive the location-basedcontent. When other American soldiers pass the house, they may receivean indication automatically, such as by seeing the virtual ‘X’ on theside of the house if they have an eyepiece or some other augmentedreality-enabled device, or by receiving a message indicating that thehouse has been cleared. In another example, content related to safetyapplications may be streamed to the eyepiece, such as alerts, targetidentification, communications, and the like.

Embodiments may provide for a way to view information associated withproducts, such as in a store. Information may include nutritionalinformation for food products, care instructions for clothing products,technical specifications for consumer electronics products, e-coupons,promotions, price comparisons with other like products, pricecomparisons with other stores, and the like. This information may beprojected in relative position with the product, to the periphery ofsight to the wearer, in relation to the store layout, and the like. Theproduct may be identified visually through a SKU, a brand tag, and thelike; transmitted by the product packaging, such as through an RFID tagon the product; transmitted by the store, such as based on the wearer'sposition in the store, in relative position to the products; and thelike.

For example, a viewer may be walking though a clothing store, and asthey walk are provided with information on the clothes on the rack,where the information is provided through the product's RFID tag. Inembodiments, the information may be delivered as a list of information,as a graphic representation, as audio and/or video presentation, and thelike. In another example, the wearer may be food shopping, andadvertisement providing facilities may be providing information to thewearer in association with products in the wearer's proximity, thewearer may be provided information when they pick up the product andview the brand, product name, SKU, and the like. In this way, the wearermay be provided a more informative environment in which to effectivelyshop.

One embodiment may allow a user to receive or share information aboutshopping or an urban area through the use of the augmented realityenabled devices such as the camera lens fitted in the eyepiece ofexemplary sunglasses. These embodiments will use augmented reality (AR)software applications such as those mentioned above in conjunction withsearching for providers of goods and services. In one scenario, thewearer of the eyepiece may walk down a street or a market for shoppingpurposes. Further, the user may activate various modes that may assistin defining user preferences for a particular scenario or environment.For example the user may enter navigation mode through which the wearermay be guided across the streets and the market for shopping of thepreferred accessories and products. The mode may be selected and variousdirections may be given by the wearer through various methods such asthrough text commands, voice commands, and the like. In an embodiment,the wearer may give a voice command to select the navigation mode whichmay result in the augmented display in front of the wearer. Theaugmented information may depict information pertinent to the locationof various shops and vendors in the market, offers in various shops andby various vendors, current happy hours, current date and time and thelike. Various sorts of options may also be displayed to the wearer. Thewearer may scroll the options and walk down the street guided throughthe navigation mode. Based on options provided, the wearer may select aplace that suits him the best for shopping based on such as offers anddiscounts and the like. In embodiments, the eyepiece may provide theability to search, browse, select, save, share, receive advertisements,and the like for items of purchase, such as viewed through the eyepiece.For example, the wearer may search for an item across the Internet andmake a purchase without making a phone call, such as through anapplication store, commerce application, and the like.

The wearer may give a voice command to navigate toward the place and thewearer may then be guided toward it. The wearer may also receiveadvertisements and offers automatically or based on request regardingcurrent deals, promotions and events in the interested location such asa nearby shopping store. The advertisements, deals and offers may appearin proximity of the wearer and options may be displayed for purchasingdesired products based on the advertisements, deals and offers. Thewearer may for example select a product and purchase it through a Googlecheckout. A message or an email may appear on the eyepiece, similar tothe one depicted in FIG. 7, with information that the transaction forthe purchase of the product has been completed. A product deliverystatus/information may also be displayed. The wearer may further conveyor alert friends and relatives regarding the offers and events throughsocial networking platforms and may also ask them to join.

In embodiments, the user may wear the head-mounted eyepiece wherein theeyepiece includes an optical assembly through which the user may view asurrounding environment and displayed content. The displayed content maycomprise one or more local advertisements. The location of the eyepiecemay be determined by an integrated location sensor and the localadvertisement may have a relevance to the location of the eyepiece. Byway of example, the user's location may be determined via GPS, RFID,manual input, and the like. Further, the user may be walking by a coffeeshop, and based on the user's proximity to the shop, an advertisement,similar to that depicted in FIG. 19, showing the store's brand 1900,such as the band for a fast food restaurant or coffee may appear in theuser's field of view. The user may experience similar types of localadvertisements as he or she moves about the surrounding environment.

In other embodiments, the eyepiece may contain a capacitive sensorcapable of sensing whether the eyepiece is in contact with human skin.Such sensor or group of sensors may be placed on the eyepiece and oreyepiece arm in such a manner that allows detection of when the glassesare being worn by a user. In other embodiments, sensors may be used todetermine whether the eyepiece is in a position such that they may beworn by a user, for example, when the earpiece is in the unfoldedposition. Furthermore, local advertisements may be sent only when theeyepiece is in contact with human skin, in a wearable position, acombination of the two, actually worn by the user and the like. In otherembodiments, the local advertisement may be sent in response to theeyepiece being powered on or in response to the eyepiece being poweredon and worn by the user and the like. By way of example, an advertisermay choose to only send local advertisements when a user is in proximityto a particular establishment and when the user is actually wearing theglasses and they are powered on allowing the advertiser to target theadvertisement to the user at the appropriate time.

In accordance with other embodiments, the local advertisement may bedisplayed to the user as a banner advertisement, two-dimensionalgraphic, text and the like. Further, the local advertisement may beassociated with a physical aspect of the user's view of the surroundingenvironment. The local advertisement may also be displayed as anaugmented reality advertisement wherein the advertisement is associatedwith a physical aspect of the surrounding environment. Suchadvertisement may be two or three-dimensional. By way of example, alocal advertisement may be associated with a physical billboard asdescribed further in FIG. 18 wherein the user's attention may be drawnto displayed content showing a beverage being poured from a billboard1800 onto an actual building in the surrounding environment. The localadvertisement may also contain sound that is displayed to the userthrough an earpiece, audio device or other means. Further, the localadvertisement may be animated in embodiments. For example, the user mayview the beverage flow from the billboard onto an adjacent building and,optionally, into the surrounding environment. Similarly, anadvertisement may display any other type of motion as desired in theadvertisement. Additionally, the local advertisement may be displayed asa three-dimensional object that may be associated with or interact withthe surrounding environment. In embodiments where the advertisement isassociated with an object in the user's view of the surroundingenvironment, the advertisement may remain associated with or inproximity to the object even as the user turns his head. For example, ifan advertisement, such as the coffee cup as described in FIG. 19, isassociated with a particular building, the coffee cup advertisement mayremain associated with and in place over the building even as the userturns his head to look at another object in his environment.

In other embodiments, local advertisements may be displayed to the userbased on a web search conducted by the user where the advertisement isdisplayed in the content of the web search results. For example, theuser may search for “happy hour” as he is walking down the street, andin the content of the search results, a local advertisement may bedisplayed advertising a local bar's beer prices.

Further, the content of the local advertisement may be determined basedon the user's personal information. The user's information may be madeavailable to a web application, an advertising facility and the like.Further, a web application, advertising facility or the user's eyepiecemay filter the advertising based on the user's personal information.Generally, for example, a user may store personal information about hislikes and dislikes and such information may be used to directadvertising to the user's eyepiece. By way of specific example, the usermay store data about his affinity for a local sports team, and asadvertisements are made available, those advertisements with hisfavorite sports team may be given preference and pushed to the user.Similarly, a user's dislikes may be used to exclude certainadvertisements from view. In various embodiments, the advertisements maybe cashed on a server where the advertisement may be accessed by atleast one of an advertising facility, web application and eyepiece anddisplayed to the user.

In various embodiments, the user may interact with any type of localadvertisement in numerous ways. The user may request additionalinformation related to a local advertisement by making at least oneaction of an eye movement, body movement and other gesture. For example,if an advertisement is displayed to the user, he may wave his hand overthe advertisement in his field of view or move his eyes over theadvertisement in order to select the particular advertisement to receivemore information relating to such advertisement. Moreover, the user maychoose to ignore the advertisement by any movement or control technologydescribed herein such as through an eye movement, body movement, othergesture and the like. Further, the user may chose to ignore theadvertisement by allowing it to be ignored by default by not selectingthe advertisement for further interaction within a given period of time.For example, if the user chooses not to gesture for more informationfrom the advertisement within five seconds of the advertisement beingdisplayed, the advertisement may be ignored by default and disappearfrom the users view. Furthermore, the user may select to not allow localadvertisements to be displayed whereby said user selects such an optionon a graphical user interface or by turning such feature off via acontrol on said eyepiece.

In other embodiments, the eyepiece may include an audio device.Accordingly, the displayed content may comprise a local advertisementand audio such that the user is also able to hear a message or othersound effects as they relate to the local advertisement. By way ofexample, and referring again to FIG. 18, while the user sees the beerbeing poured, he will actually be able to hear an audio transmissioncorresponding to the actions in the advertisement. In this case, theuser may hear the bottle open and then the sound of the liquid pouringout of the bottle and onto the rooftop. In yet other embodiments, adescriptive message may be played, and or general information may begiven as part of the advertisement. In embodiments, any audio may beplayed as desired for the advertisement.

In accordance with another embodiment, social networking may befacilitated with the use of the augmented reality enabled devices suchas a camera lens fitted in the eyepiece. This may be utilized to connectseveral users or other persons that may not have the augmented realityenabled device together who may share thoughts and ideas with eachother. For instance, the wearer of the eyepiece may be sitting in aschool campus along with other students. The wearer may connect with andsend a message to a first student who may be present in a coffee shop.The wearer may ask the first student regarding persons interested in aparticular subject such as environmental economics for example. As otherstudents pass through the field of view of the wearer, the camera lensfitted inside the eyepiece may track and match the students to anetworking database such as ‘Google me’ that may contain publicprofiles. Profiles of interested and relevant persons from the publicdatabase may appear and pop-up in front of the wearer on the eyepiece.Some of the profiles that may not be relevant may either be blocked orappear blocked to the user. The relevant profiles may be highlighted forquick reference of the wearer. The relevant profiles selected by thewearer may be interested in the subject environmental economics and thewearer may also connect with them. Further, they may also be connectedwith the first student. In this manner, a social network may beestablished by the wearer with the use of the eyepiece enabled with thefeature of the augmented reality. The social networks managed by thewearer and the conversations therein may be saved for future reference.

The present disclosure may be applied in a real estate scenario with theuse of the augmented reality enabled devices such as a camera lensfitted in an eyepiece. The wearer, in accordance with this embodiment,may want to get information about a place in which the user may bepresent at a particular time such as during driving, walking, joggingand the like. The wearer may, for instance, want to understandresidential benefits and loss in that place. He may also want to getdetailed information about the facilities in that place. Therefore, thewearer may utilize a map such as a Google online map and recognize thereal estate that may be available there for lease or purchase. As notedabove, the user may receive information about real estate for sale orrent using mobile Internet applications such as Layar. In one suchapplication, information about buildings within the user's field of viewis projected onto the inside of the glasses for consideration by theuser. Options may be displayed to the wearer on the eyepiece lens forscrolling, such as with a trackpad mounted on a frame of the glasses.The wearer may select and receive information about the selected option.The augmented reality enabled scenes of the selected options may bedisplayed to the wearer and the wearer may be able to view pictures andtake a facility tour in the virtual environment. The wearer may furtherreceive information about real estate agents and fix an appointment withone of those. An email notification or a call notification may also bereceived on the eyepiece for confirmation of the appointment. If thewearer finds the selected real estate of worth, a deal may be made andthat may be purchased by the wearer.

In accordance with another embodiment, customized and sponsored toursand travels may be enhanced through the use of the augmentedreality-enabled devices, such as a camera lens fitted in the eyepiece.For instance, the wearer (as a tourist) may arrive in a city such asParis and wants to receive tourism and sightseeing related informationabout the place to accordingly plan his visit for the consecutive daysduring his stay. The wearer may put on his eyepiece or operate any otheraugmented reality enabled device and give a voice or text commandregarding his request. The augmented reality enabled eyepiece may locatewearer position through geo-sensing techniques and decide tourismpreferences of the wearer. The eyepiece may receive and displaycustomized information based on the request of the wearer on a screen.The customized tourism information may include information about artgalleries and museums, monuments and historical places, shoppingcomplexes, entertainment and nightlife spots, restaurants and bars, mostpopular tourist destinations and centers/attractions of tourism, mostpopular local/cultural/regional destinations and attractions, and thelike without limitations. Based on user selection of one or more ofthese categories, the eyepiece may prompt the user with other questionssuch as time of stay, investment in tourism and the like. The wearer mayrespond through the voice command and in return receive customized tourinformation in an order as selected by the wearer. For example thewearer may give a priority to the art galleries over monuments.Accordingly, the information may be made available to the wearer.Further, a map may also appear in front of the wearer with differentsets of tour options and with different priority rank such as:

Priority Rank 1: First tour Option (Champs Elyse, Louvre, Rodin, Museum,Famous Café)

Priority Rank 2: Second option

Priority Rank 3: Third Option

The wearer, for instance, may select the first option since it is rankedas highest in priority based on wearer indicated preferences.Advertisements related to sponsors may pop up right after selection.Subsequently, a virtual tour may begin in the augmented reality mannerthat may be very close to the real environment. The wearer may forexample take a 30 seconds tour to a vacation special to the AtlantisResort in the Bahamas. The virtual 3D tour may include a quick look atthe rooms, beach, public spaces, parks, facilities, and the like. Thewearer may also experience shopping facilities in the area and receiveoffers and discounts in those places and shops. At the end of the day,the wearer might have experienced a whole day tour sitting in hischamber or hotel. Finally, the wearer may decide and schedule his planaccordingly.

Another embodiment may allow information concerning auto repairs andmaintenance services with the use of the augmented reality enableddevices such as a camera lens fitted in the eyepiece. The wearer mayreceive advertisements related to auto repair shops and dealers bysending a voice command for the request. The request may, for exampleinclude a requirement of oil change in the vehicle/car. The eyepiece mayreceive information from the repair shop and display to the wearer. Theeyepiece may pull up a 3D model of the wearer's vehicle and show theamount of oil left in the car through an augmented reality enabledscene/view. The eyepiece may show other relevant information also aboutthe vehicle of the wearer such as maintenance requirements in otherparts like brake pads. The wearer may see 3D view of the wearing brakepads and may be interested in getting those repaired or changed.Accordingly, the wearer may schedule an appointment with a vendor to fixthe problem via using the integrated wireless communication capabilityof the eyepiece. The confirmation may be received through an email or anincoming call alert on the eyepiece camera lens.

In accordance with another embodiment, gift shopping may benefit throughthe use of the augmented reality enabled devices such as a camera lensfitted in the eyepiece. The wearer may post a request for a gift forsome occasion through a text or voice command. The eyepiece may promptthe wearer to answer his preferences such as type of gifts, age group ofthe person to receive the gift, cost range of the gift and the like.Various options may be presented to the user based on the receivedpreferences. For instance, the options presented to the wearer may be:Cookie basket, Wine and cheese basket, Chocolate assortment, Golfer'sgift basket, and the like.

The available options may be scrolled by the wearer and the best fitoption may be selected via the voice command or text command. Forexample, the wearer may select the Golfer's gift basket. A 3D view ofthe Golfer's gift basket along with a golf course may appear in front ofthe wearer. The virtual 3D view of the Golfer's gift basket and the golfcourse enabled through the augmented reality may be perceived very closeto the real world environment. The wearer may finally respond to theaddress, location and other similar queries prompted through theeyepiece. A confirmation may then be received through an email or anincoming call alert on the eyepiece camera lens.

Another application that may appeal to users is mobile on-line gamingusing the augmented reality glasses. These games may be computer videogames, such as those furnished by Electronic Arts Mobile, UbiSoft andActivision Blizzard, e.g., World of Warcraft® (WoW). Just as games andrecreational applications are played on computers at home (rather thancomputers at work), augmented reality glasses may also use gamingapplications. The screen may appear on an inside of the glasses so thata user may observe the game and participate in the game. In addition,controls for playing the game may be provided through a virtual gamecontroller, such as a joystick, control module or mouse, describedelsewhere herein. The game controller may include sensors or otheroutput type elements attached to the user's hand, such as for feedbackfrom the user through acceleration, vibration, force, pressure,electrical impulse, temperature, electric field sensing, and the like.Sensors and actuators may be attached to the user's hand by way of awrap, ring, pad, glove, bracelet, and the like. As such, an eyepiecevirtual mouse may allow the user to translate motions of the hand,wrist, and/or fingers into motions of the cursor on the eyepiecedisplay, where “motions” may include slow movements, rapid motions,jerky motions, position, change in position, and the like, and may allowusers to work in three dimensions, without the need for a physicalsurface, and including some or all of the six degrees of freedom.

As seen in FIG. 27, gaming application implementations 2700 may use boththe internet and a GPS. In one embodiment, a game is downloaded from acustomer database via a game provider, perhaps using their web servicesand the internet as shown, to a user computer or augmented realityglasses. At the same time, the glasses, which also havetelecommunication capabilities, receive and send telecommunications andtelemetry signals via a cellular tower and a satellite. Thus, an on-linegaming system has access to information about the user's location aswell as the user's desired gaming activities.

Games may take advantage of this knowledge of the location of eachplayer. For example, the games may build in features that use theplayer's location, via a GPS locator or magnetometer locator, to awardpoints for reaching the location. The game may also send a message,e.g., display a clue, or a scene or images, when a player reaches aparticular location. A message, for example, may be to go to a nextdestination, which is then provided to the player. Scenes or images maybe provided as part of a struggle or an obstacle which must be overcome,or as an opportunity to earn game points. Thus, in one embodiment,augmented reality eyepieces or glasses may use the wearer's location toquicken and enliven computer-based video games.

One method of playing augmented reality games is depicted in FIG. 28. Inthis method 2800, a user logs into a website whereby access to a game ispermitted. The game is selected. In one example, the user may join agame, if multiple player games are available and desired; alternatively,the user may create a custom game, perhaps using special roles the userdesired. The game may be scheduled, and in some instances, players mayselect a particular time and place for the game, distribute directionsto the site where the game will be played, etc. Later, the players meetand check into the game, with one or more players using the augmentedreality glasses. Participants then play the game and if applicable, thegame results and any statistics (scores of the players, game times,etc.) may be stored. Once the game has begun, the location may changefor different players in the game, sending one player to one locationand another player or players to a different location. The game may thenhave different scenarios for each player or group of players, based ontheir GPS or magnetometer-provided locations. Each player may also besent different messages or images based on his or her role, his or herlocation, or both. Of course, each scenario may then lead to othersituations, other interactions, directions to other locations, and soforth. In one sense, such a game mixes the reality of the player'slocation with the game in which the player is participating.

Games can range from simple games of the type that would be played in apalm of a player's hand, such as small, single player games.Alternatively, more complicated, multi-player games may also be played.In the former category are games such as SkySiege, AR Drone and FireFighter 360. In addition, multiplayer games are also easily envisioned.Since all players must log into the game, a particular game may beplayed by friends who log in and specify the other person or persons.The location of the players is also available, via GPS or other method.Sensors in the augmented reality glasses or in a game controller asdescribed above, such as accelerometers, gyroscopes or even a magneticcompass, may also be used for orientation and game playing. An exampleis AR Invaders, available for iPhone applications from the App Store.Other games may be obtained from other vendors and for non-iPhone typesystems, such as Layar, of Amsterdam and Paris SA, Paris, France,supplier of AR Drone, AR Flying Ace and AR Pursuit.

In embodiments, games may also be in 3D such that the user canexperience 3D gaming. For example, when playing a 3D game, the user mayview a virtual, augmented reality or other environment where the user isable to control his view perspective. The user may turn his head to viewvarious aspects of the virtual environment or other environment. Assuch, when the user turns his head or makes other movements, he may viewthe game environment as if he were actually in such environment. Forexample, the perspective of the user may be such that the user is put‘into’ a 3D game environment with at least some control over the viewingperspective where the user may be able to move his head and have theview of the game environment change in correspondence to the changedhead position. Further, the user may be able to ‘walk into’ the gamewhen he physically walks forward, and have the perspective change as theuser moves. Further, the perspective may also change as the user movesthe gazing view of his eyes, and the like. Additional image informationmay be provided, such as at the sides of the user's view that could beaccessed by turning the head.

In embodiments, the 3D game environment may be projected onto the lensesof the glasses or viewed by other means. Further, the lenses may beopaque or transparent. In embodiments, the 3D game image may beassociated with and incorporate the external environment of the usersuch that the user may be able to turn his head and the 3D image andexternal environment stay together. Further, such 3D gaming image andexternal environment associations may change such that the 3D imageassociates with more than one object or more than one part of an objectin the external environment at various instances such that it appears tothe user that the 3D image is interacting with various aspects orobjects of the actual environment. By way of example, the user may viewa 3D game monster climb up a building or on to an automobile where suchbuilding or automobile is an actual object in the user's environment. Insuch a game, the user may interact with the monster as part of the 3Dgaming experience. The actual environment around the user may be part ofthe 3D gaming experience. In embodiments where the lenses aretransparent, the user may interact in a 3D gaming environment whilemoving about his or her actual environment. The 3D game may incorporateelements of the user's environment into the game, it may be whollyfabricated by the game, or it may be a mixture of both.

In embodiments, the 3D images may be associated with or generated by anaugmented reality program, 3D game software and the like or by othermeans. In embodiments where augmented reality is employed for thepurpose of 3D gaming, a 3D image may appear or be perceived by the userbased on the user's location or other data. Such an augmented realityapplication may provide for the user to interact with such 3D image orimages to provide a 3D gaming environment when using the glasses. As theuser changes his location, for example, play in the game may advance andvarious 3D elements of the game may become accessible or inaccessible tothe viewer. By way of example, various 3D enemies of the user's gamecharacter may appear in the game based on the actual location of theuser. The user may interact with or cause reactions from other usersplaying the game and or 3D elements associated with the other usersplaying the game. Such elements associated with users may includeweapons, messages, currency, a 3D image of the user and the like. Basedon a user's location or other data, he or she may encounter, view, orengage, by any means, other users and 3D elements associated with otherusers. In embodiments, 3D gaming may also be provided by softwareinstalled in or downloaded to the glasses where the user's location isor is not used.

In embodiments, the lenses may be opaque to provide the user with avirtual reality or other virtual 3D gaming experience where the user is‘put into’ the game where the user's movements may change the viewingperspective of the 3D gaming environment for the user. The user may movethrough or explore the virtual environment through various body, head,and or eye movements, use of game controllers, one or more touchscreens, or any of the control techniques described herein which mayallow the user to navigate, manipulate, and interact with the 3Denvironment, and thereby play the 3D game.

In various embodiments, the user may navigate, interact with andmanipulate the 3D game environment and experience 3D gaming via body,hand, finger, eye, or other movements, through the use of one or morewired or wireless controllers, one or more touch screens, any of thecontrol techniques described herein, and the like.

In embodiments, internal and external facilities available to theeyepiece may provide for learning the behavior of a user of theeyepiece, and storing that learned behavior in a behavioral database toenable location-aware control, activity-aware control, predictivecontrol, and the like. For example, a user may have events and/ortracking of actions recorded by the eyepiece, such as commands from theuser, images sensed through a camera, GPS location of the user, sensorinputs over time, triggered actions by the user, communications to andfrom the user, user requests, web activity, music listened to,directions requested, recommendations used or provided, and the like.This behavioral data may be stored in a behavioral database, such astagged with a user identifier or autonomously. The eyepiece may collectthis data in a learn mode, collection mode, and the like. The eyepiecemay utilize past data taken by the user to inform or remind the user ofwhat they did before, or alternatively, the eyepiece may utilize thedata to predict what eyepiece functions and applications the user mayneed based on past collected experiences. In this way, the eyepiece mayact as an automated assistant to the user, for example, launchingapplications at the usual time the user launches them, turning offaugmented reality and the GPS when nearing a location or entering abuilding, streaming in music when the user enters the gym, and the like.Alternately, the learned behavior and/or actions of a plurality ofeyepiece users may be autonomously stored in a collective behaviordatabase, where learned behaviors amongst the plurality of users areavailable to individual users based on similar conditions. For example,a user may be visiting a city, and waiting for a train on a platform,and the eyepiece of the user accesses the collective behavior databaseto determine what other users have done while waiting for the train,such as getting directions, searching for points of interest, listeningto certain music, looking up the train schedule, contacting the citywebsite for travel information, connecting to social networking sitesfor entertainment in the area, and the like. In this way, the eyepiecemay be able to provide the user with an automated assistant with thebenefit of many different user experiences. In embodiments, the learnedbehavior may be used to develop preference profiles, recommendations,advertisement targeting, social network contacts, behavior profiles forthe user or groups of users, and the like, for/to the user.

In an embodiment, the augmented reality eyepiece or glasses may includeone or more acoustic sensors for detecting sound 2900. An example isdepicted above in FIG. 29. In one sense, acoustic sensors are similar tomicrophones, in that they detect sounds. Acoustic sensors typically haveone or more frequency bandwidths at which they are more sensitive, andthe sensors can thus be chosen for the intended application. Acousticsensors are available from a variety of manufacturers and are availablewith appropriate transducers and other required circuitry. Manufacturersinclude ITT Electronic Systems, Salt Lake City, Utah, USA; MeggittSensing Systems, San Juan Capistrano, Calif., USA; and NationalInstruments, Austin, Tex., USA. Suitable microphones include those whichcomprise a single microphone as well as those which comprise an array ofmicrophones, or a microphone array.

Acoustic sensors may include those using micro electromechanical systems(MEMS) technology. Because of the very fine structure in a MEMS sensor,the sensor is extremely sensitive and typically has a wide range ofsensitivity. MEMS sensors are typically made using semiconductormanufacturing techniques. An element of a typical MEMS accelerometer isa moving beam structure composed of two sets of fingers. One set isfixed to a solid ground plane on a substrate; the other set is attachedto a known mass mounted on springs that can move in response to anapplied acceleration. This applied acceleration changes the capacitancebetween the fixed and moving beam fingers. The result is a verysensitive sensor. Such sensors are made, for example, bySTMicroelectronics, Austin, Tex. and Honeywell International, MorristownN.J., USA.

In addition to identification, sound capabilities of the augmentedreality devices may also be applied to locating an origin of a sound. Asis well known, at least two sound or acoustic sensors are needed tolocate a sound. The acoustic sensor will be equipped with appropriatetransducers and signal processing circuits, such as a digital signalprocessor, for interpreting the signal and accomplishing a desired goal.One application for sound locating sensors may be to determine theorigin of sounds from within an emergency location, such as a burningbuilding, an automobile accident, and the like. Emergency workersequipped with embodiments described herein may each have one or morethan one acoustic sensors or microphones embedded within the frame. Ofcourse, the sensors could also be worn on the person's clothing or evenattached to the person. In any event, the signals are transmitted to thecontroller of the augmented reality eyepiece. The eyepiece or glassesare equipped with GPS technology and may also be equipped withdirection-finding capabilities; alternatively, with two sensors perperson, the microcontroller can determine a direction from which thenoise originated.

If there are two or more firefighters, or other emergency responders,their location is known from their GPS capabilities. Either of the two,or a fire chief, or the control headquarters, then knows the position oftwo responders and the direction from each responder to the detectednoise. The exact point of origin of the noise can then be determinedusing known techniques and algorithms. See e.g., Acoustic Vector-SensorBeamforming and Capon Direction Estimation, M. Hawkes and A. Nehorai,IEEE Transactions on Signal Processing, vol. 46, no. 9, September 1998,at 2291-2304; see also Cramér-Rao Bounds for Direction Finding by anAcoustic Vector Sensor Under Nonideal Gain-Phase Responses,Noncollocation or Nonorthogonal Orientation, P. K. Tam and K. T. Wong,IEEE Sensors Journal, vol. 9. No. 8, August 2009, at 969-982. Thetechniques used may include timing differences (differences in time ofarrival of the parameter sensed), acoustic velocity differences, andsound pressure differences. Of course, acoustic sensors typicallymeasure levels of sound pressure (e.g., in decibels), and these otherparameters may be used in appropriate types of acoustic sensors,including acoustic emission sensors and ultrasonic sensors ortransducers.

The appropriate algorithms and all other necessary programming may bestored in the microcontroller of the eyepiece, or in memory accessibleto the eyepiece. Using more than one responder, or several responders, alikely location may then be determined, and the responders can attemptto locate the person to be rescued. In other applications, respondersmay use these acoustic capabilities to determine the location of aperson of interest to law enforcement. In still other applications, anumber of people on maneuvers may encounter hostile fire, includingdirect fire (line of sight) or indirect fire (out of line of sight,including high angle fire). The same techniques described here may beused to estimate a location of the hostile fire. If there are severalpersons in the area, the estimation may be more accurate, especially ifthe persons are separated at least to some extent, over a wider area.This may be an effective tool to direct counter-battery orcounter-mortar fire against hostiles. Direct fire may also be used ifthe target is sufficiently close.

An example using embodiments of the augmented reality eyepieces isdepicted in FIG. 29B. In this example 2900B, numerous soldiers are onpatrol, each equipped with augmented reality eyepieces, and are alertfor hostile fire. The sounds detected by their acoustic sensors ormicrophones may be relayed to a squad vehicle as shown, to their platoonleader, or to a remote tactical operations center (TOC) or command post(CP). Alternatively, or in addition to these, the signals may also besent to a mobile device, such as an airborne platform, as shown.Communications among the soldiers and the additional locations may befacilitated using a local area network, or other network. In addition,all the transmitted signals may be protected by encryption or otherprotective measures. One or more of the squad vehicle, the platooncommander, the mobile platform, the TOC or the CP will have anintegration capability for combining the inputs from the severalsoldiers and determining a possible location of the hostile fire. Thesignals from each soldier will include the location of the soldier froma GPS capability inherent in the augmented reality glasses or eyepiece.The acoustic sensors on each soldier may indicate a possible directionof the noise. Using signals from several soldiers, the direction andpossibly the location of the hostile fire may be determined. Thesoldiers may then neutralize the location.

In addition to microphones, the augmented reality eyepiece may beequipped with ear buds, which may be articulating ear buds, as mentionedelse where herein, and may be removably attached 1403, or may beequipped with an audio output jack 1401. The eyepiece and ear buds maybe equipped to deliver noise-cancelling interference, allowing the userto better hear sounds delivered from the audio-video communicationscapabilities of the augmented reality eyepiece or glasses, and mayfeature automatic gain control. The speakers or ear buds of theaugmented reality eyepiece may also connect with the full audio andvisual capabilities of the device, with the ability to deliver highquality and clear sound from the included telecommunications device. Asnoted elsewhere herein, this includes radio or cellular telephone (smartphone) audio capabilities, and may also include complementarytechnologies, such as Bluetooth™ capabilities or related technologies,such as IEEE 802.11, for wireless personal area networks (WPAN).

Another aspect of the augmented audio capabilities includes speechrecognition and identification capabilities. Speech recognition concernsunderstanding what is said while speech identification concernsunderstanding who the speaker is. Speech identification may work hand inhand with the facial recognition capabilities of these devices to morepositively identify persons of interest. As described elsewhere in thisdocument, a camera connected as part of the augmented reality eyepiececan unobtrusively focus on desired personnel, such as a single person ina crowd or multiple faces in a crowd. Using the camera and appropriatefacial recognition software, an image of the person or people may betaken. The features of the image are then broken down into any number ofmeasurements and statistics, and the results are compared to a databaseof known persons. An identity may then be made. In the same manner, avoice or voice sampling from the person of interest may be taken. Thesample may be marked or tagged, e.g., at a particular time interval, andlabeled, e.g., a description of the person's physical characteristics ora number. The voice sample may be compared to a database of knownpersons, and if the person's voice matches, then an identification maybe made. In embodiments, multiple individuals of interest may byselected, such as for biometric identification. The multiple selectionmay be through the use of a cursor, a hand gesture, an eye movement, andthe like. As a result of the multiple selection, information concerningthe selected individuals may be provided to the user, such as throughthe display, through audio, and the like.

In embodiments where the camera is used for biometric identification ofmultiple people in a crowd, control technologies described herein may beused to select faces or irises for imaging. For example, a cursorselection using the hand-worn control device may be used to selectmultiple faces in a view of the user's surrounding environment. Inanother example, gaze tracking may be used to select which faces toselect for biometric identification. In another example, the hand-worncontrol device may sense a gesture used to select the individuals, suchas pointing at each individual.

In one embodiment, important characteristics of a particular person'sspeech may be understood from a sample or from many samples of theperson's voice. The samples are typically broken into segments, framesand subframes. Typically, important characteristics include afundamental frequency of the person's voice, energy, formants, speakingrate, and the like. These characteristics are analyzed by software whichanalyses the voice according to certain formulae or algorithms. Thisfield is constantly changing and improving. However, currently suchclassifiers may include algorithms such as neural network classifiers,k-classifiers, hidden Markov models, Gaussian mixture models and patternmatching algorithms, among others.

A general template 3100 for speech recognition and speakeridentification is depicted in FIG. 31. A first step 3101 is to provide aspeech signal. Ideally, one has a known sample from prior encounterswith which to compare the signal. The signal is then digitized in step3102 and is partitioned in step 3103 into fragments, such as segments,frames and subframes. Features and statistics of the speech sample arethen generated and extracted in step 3104. The classifier, or more thanone classifier, is then applied in step 3105 to determine generalclassifications of the sample. Post-processing of the sample may then beapplied in step 3106, e.g., to compare the sample to known samples forpossible matching and identification. The results may then be output instep 3107. The output may be directed to the person requesting thematching, and may also be recorded and sent to other persons and to oneor more databases.

In an embodiment, the audio capabilities of the eyepiece include hearingprotection with the associated earbuds. The audio processor of theeyepiece may enable automatic noise suppression, such as if a loud noiseis detected near the wearer's head. Any of the control technologiesdescribed herein may be used with automatic noise suppression.

In an embodiment, the eyepiece may include a nitinol head strap. Thehead strap may be a thin band of curved metal which may either pull outfrom the arms of the eyepiece or rotate out and extend out to behind thehead to secure the eyepiece to the head. In one embodiment, the tip ofthe nitinol strap may have a silicone cover such that the silicone coveris grasped to pull out from the ends of the arms. In embodiments, onlyone arm has a nitinol band, and it gets secured to the other arm to forma strap. In other embodiments, both arms have a nitinol band and bothsides get pulled out to either get joined to form a strap orindependently grasp a portion of the head to secure the eyepiece on thewearer's head. In embodiments, the eyepiece may have interchangeableequipment to attach the eyepiece to an individual's head, such as ajoint where a head strap, glasses arms, helmet strap, helmet snapconnection, and the like may be attached. For example, there may be ajoint in the eyepiece near the user's temple where the eyepiece mayattach to a strap, and where the strap may be disconnected so the usermay attach arms to make the eyepiece take the form of glasses, attach toa helmet, and the like. In embodiments, the interchangeable equipmentattaching the eyepiece to the user's head or to a helmet may include anembedded antenna. For example, a Nitinol head strap may have an embeddedantenna inside, such as for a particular frequency, for a plurality offrequencies, and the like. In addition, the arm, strap, and the like,may contain RF absorbing foam in order to aid in the absorption of RFenergy while the antenna is used in transmission.

Referring to FIG. 21, the eyepiece may include one or more adjustablewrap around extendable arms 2134. The adjustable wrap around extendablearms 2134 may secure the position of the eyepiece to the user's head.One or more of the extendable arms 2134 may be made out of a shapememory material. In embodiments, one or both of the arms may be made ofnitinol and/or any shape-memory material. In other instances, the end ofat least one of the wrap around extendable arms 2134 may be covered withsilicone. Further, the adjustable wrap around extendable arms 2134 mayextend from the end of an eyepiece arm 2116. They may extendtelescopically and/or they may slide out from an end of the eyepiecearms. They may slide out from the interior of the eyepiece arms 2116 orthey may slide along an exterior surface of the eyepiece arms 2116.Further, the extendable arms 2134 may meet and secure to each other. Theextendable arms may also attach to another portion of the head mountedeyepiece to create a means for securing the eyepiece to the user's head.The wrap around extendable arms 2134 may meet to secure to each other,interlock, connect, magnetically couple, or secure by other means so asto provide a secure attachment to the user's head. In embodiments, theadjustable wrap around extendable arms 2134 may also be independentlyadjusted to attach to or grasp portions of the user's head. As such theindependently adjustable arms may allow the user increasedcustomizability for a personalized fit to secure the eyepiece to theuser's head. Further, in embodiments, at least one of the wrap aroundextendable arms 2134 may be detachable from the head mounted eyepiece.In yet other embodiments, the wrap around extendable arms 2134 may be anadd-on feature of the head mounted eyepiece. In such instances, the usermay chose to put extendable, non-extendable or other arms on to the headmounted eyepiece. For example, the arms may be sold as a kit or part ofa kit that allows the user to customize the eyepiece to his or herspecific preferences. Accordingly, the user may customize that type ofmaterial from which the adjustable wrap around extendable arm 2134 ismade by selecting a different kit with specific extendable arms suitedto his preferences. Accordingly, the user may customize his eyepiece forhis particular needs and preferences.

In yet other embodiments, an adjustable strap, 2142, may be attached tothe eyepiece arms such that it extends around the back of the user'shead in order to secure the eyepiece in place. The strap may be adjustedto a proper fit. It may be made out of any suitable material, includingbut not limited to rubber, silicone, plastic, cotton and the like.

In an embodiment, the eyepiece may be secured to the user's head by aplurality of other structures, such a rigid arm, a flexible arm, agooseneck flex arm, a cable tensioned system, and the like. Forinstance, a flexible arm may be constructed from a flexible tubing, suchas in a gooseneck configuration, where the flexible arm may be flexedinto position to adjust to the fit of a given user, and where theflexible arm may be reshaped as needed. In another instance, a flexiblearm may be constructed from a cable tensioned system, such as in arobotic finger configuration, having multiple joints connecting membersthat are bent into a curved shape with a pulling force applied to acable running through the joints and members. In this case, thecable-driven system may implement an articulating ear horn for sizeadjustment and eyepiece headwear retention. The cable-tensioned systemmay have two or more linkages, the cable may be stainless steel,Nitinol-based, electro-actuated, ratcheted, wheel adjusted, and thelike.

In an embodiment, the eyepiece may include security features, such asM-Shield Security, Secure content, DSM, Secure Runtime, IPSec, and thelike. Other software features may include: User Interface, Apps,Framework, BSP, Codecs, Integration, Testing, System Validation, and thelike.

In an embodiment, the eyepiece materials may be chosen to enableruggedization.

In an embodiment, the eyepiece may be able to access a 3G access pointthat includes a 3G radio, an 802.11b connection and a Bluetoothconnection to enable hopping data from a device to a 3G-enableembodiment of the eyepiece.

The present disclosure also relates to methods and apparatus for thecapture of biometric data about individuals. The methods and apparatusprovide wireless capture of fingerprints, iris patterns, facialstructure and other unique biometric features of individuals and thensend the data to a network or directly to the eyepiece. Data collectedfrom an individual may also be compared with previously collected dataand used to identify a particular individual.

In embodiments, the eyepiece 100 may be associated with mobile biometricdevices, such as a biometric flashlight 7300, a biometric phone 5000, abiometric camera, a pocket biometric device 5400, an arm strap biometricdevice 5600, and the like, where the mobile biometrics device may act asa stand-alone device or in communications with the eyepiece, such as forcontrol of the device, display of data from the device, storage of data,linking to an external system, linking to other eyepieces and/or othermobile biometrics devices, and the like. The mobile biometrics devicemay enable a soldier or other non-military personnel to collect orutilize existing biometrics to profile an individual. The device mayprovide for tracking, monitoring, and collecting biometric records suchas including video, voice, gait, face, iris biometrics and the like. Thedevice may provide for geo-location tags for collected data, such aswith time, date, location, data-taking personnel, the environment, andthe like. The device may be able to capture and record fingerprints,palm prints, scars, marks, tattoos, audio, video, annotations, and thelike, such as utilizing a thin film sensor, recording, collecting,identifying, and verifying face, fingerprint, iris, latent fingerprints,latent palm prints, voice, pocket litter, and other identifying visiblemarks and environmental data. The device may be able to read prints wetor dry. The device may include a camera, such as with, IR illumination,UV illumination, and the like, with a capability to see through, dust,smoke, haze, and the like. The camera may support dynamic rangeextension, adaptive defect pixel correction, advanced sharpnessenhancement, geometric distortion correction, advanced color management,hardware-based face detection, video stabilization, and the like. Inembodiments, the camera output may be transmitted to the eyepiece forpresentation to the soldier. The device may accommodate a plurality ofother sensors, such as described herein, including an accelerometer,compass, ambient light, proximity, barometric and temperature sensors,and the like, depending on requirements. The device may also have amosaic print sensor, as described herein, producing high resolutionimages of the whorls and pores of an individual's fingerprint, multiplefinger prints simultaneously, palm print, and the like. A soldier mayutilize a mobile biometrics device to more easily collect personnelinformation, such as for document and media exploitation (DOMEX). Forinstance, during an interview, enrollment, interrogations, and the like,operators may photograph and read identifying data or ‘pocket litter’(e.g. passport, ID cards, personal documents, cell phone directories,pictures), take biometric data, and the like, into a person of interestprofile that may be entered into a searchable secure database. Inembodiments, biometric data may be filed using the most salient imageplus manual entry, enabling partial data capture. Data may beautomatically geo-located, time/date stamped, filed into a digitaldossier, and the like, such as with a locally or network assigned globalunique identifier (GUID). For instance, a face image may be captured atthe scene of an IED bombing, the left iris image may be captured at ascene of a suicide bombing, latent fingerprints may be lifted from asniper rifle, each taken from a different mobile biometrics device atdifferent locations and times, and together identifying a person ofinterest from the multiple inputs, such as at a random vehicleinspection point.

A further embodiment of the eyepiece may be used to provide biometricdata collection and result reporting. Biometric data may be visualbiometric data, such as facial biometric data or iris biometric data, ormay be audio biometric data. FIG. 39 depicts an embodiment providingbiometric data capture. The assembly, 3900 incorporates the eyepiece100, discussed above in connection with FIG. 1. Eyepiece 100 provides aninteractive head-mounted eyepiece that includes an optical assembly.Other eyepieces providing similar functionality may also be used.Eyepieces may also incorporate global positioning system capability topermit location information display and reporting.

The optical assembly allows a user to view the surrounding environment,including individuals in the vicinity of the wearer. An embodiment ofthe eyepiece allows a user to biometrically identify nearby individualsusing facial images and iris images or both facial and iris images oraudio samples. The eyepiece incorporates a corrective element thatcorrects a user's view of the surrounding environment and also displayscontent provided to the user through in integrated processor and imagesource. The integrated image source introduces the content to bedisplayed to the user to the optical assembly.

The eyepiece also includes an optical sensor for capturing biometricdata. The integrated optical sensor, in an embodiment may incorporate acamera mounted on the eyepiece. This camera is used to capture biometricimages of an individual near the user of the eyepiece. The user directsthe optical sensor or the camera toward a nearby individual bypositioning the eyepiece in the appropriate direction, which may be donejust by looking at the individual. The user may select whether tocapture one or more of a facial image, an iris image, or an audiosample.

The biometric data that may be captured by the eyepiece illustrated inFIG. 39 includes facial images for facial recognition, iris images foriris recognition, and audio samples for voice identification. Theeyepiece 3900 incorporates multiple microphones 3902 in an endfire arraydisposed along both the right and left temples of the eyepiece. Themicrophone arrays 3902 are specifically tuned to enable capture of humanvoices in an environment with a high level of ambient noise. Themicrophones may be directional, steerable, and covert. Microphones 3902provide selectable options for improved audio capture, includingomni-directional operation, or directional beam operation. Directionalbeam operation allows a user to record audio samples from a specificindividual by steering the microphone array in the direction of thesubject individual. Adaptive microphone arrays may be created that willallow the operator to steer the directionality of the microphone arrayin three dimensions, where the directional beam may be adjusted in realtime to maximize signal or minimize interfering noise for a nonstationary target. Array processing may allow summing of cardioidelements by analog or digital means, where there may be switchingbetween omni and directional array operations. In embodiments, beamforming, array steering, adaptive array processing (speech sourcelocation), and the like, may be performed by the on-board processor. Inan embodiment, the microphone may be capable of 10 dB directionalrecording.

Audio biometric capture is enhanced by incorporating phased array audioand video tracking for audio and video capture. Audio tracking allowsfor continuing to capture an audio sample when the target individual ismoving in an environment with other noise sources. In embodiments, theuser's voice may be subtracted from the audio track so as to enable aclearer rendition of the target individual, such as for distinguishingwhat is being said, to provide better location tracking, to providebetter audio tracking, and the like.

To provide power for the display optics and biometric data collectionthe eyepiece 3900 also incorporates a lithium-ion battery 3904, that iscapable of operating for over twelve hours on a single charge. Inaddition, the eyepiece 100 also incorporates a processor and solid-statememory 3906 for processing the captured biometric data. The processorand memory are configurable to function with any software or algorithmused as part of a biometric capture protocol or format, such as the .wavformat.

A further embodiment of the eyepiece assembly 3900 provides anintegrated communications facility that transmits the captured biometricdata to a remote facility that stores the biometric data in a biometricdata database. The biometric data database interprets the capturedbiometric data, interprets the data, and prepares content for display onthe eyepiece.

In operation, a wearer of the eyepiece desiring to capture biometricdata from a nearby observed individual positions himself or herself sothat the individual appears in the field of view of the eyepiece. Oncein position the user initiates capture of biometric information.Biometric information that may be captured includes iris images, facialimages, and audio data.

In operation, a wearer of the eyepiece desiring to capture audiobiometric data from a nearby observed individual positions himself orherself so that the individual appears is near the eyepiece,specifically, near the microphone arrays located in the eyepiecetemples. Once in position the user initiates capture of audio biometricinformation. This audio biometric information consists of a recordedsample of the target individual speaking. Audio samples may be capturedin conjunction with visual biometric data, such as iris and facialimages.

To capture an iris image, the wearer/user observes the desiredindividual and positions the eyepiece such that the optical sensorassembly or camera may collect an image of the biometric parameters ofthe desired individual. Once captured the eyepiece processor andsolid-state memory prepare the captured image for transmission to theremote computing facility for further processing.

The remote computing facility receives the transmitted biometric imageand compares the transmitted image to previously captured biometric dataof the same type. Iris or facial images are compared with previouslycollected iris or facial images to determine if the individual has beenpreviously encountered and identified.

Once the comparison has been made, the remote computing facilitytransmits a report of the comparison to the wearer/user's eyepiece, fordisplay. The report may indicate that the captured biometric imagematches previously captured images. In such cases, the user receives areport including the identity of the individual, along with otheridentifying information or statistics. Not all captured biometric dataallows for an unambiguous determination of identity. In such cases, theremote computing facility provides a report of findings and may requestthe user to collect additional biometric data, possibly of a differenttype, to aid in the identification and comparison process. Visualbiometric data may be supplemented with audio biometric data as afurther aid to identification.

Facial images are captured in a similar manner as iris images. The fieldof view is necessarily larger, due to the size of the images collected.This also permits to user to stand further off from the subject whosefacial biometric data is being captured.

In operation the user may have originally captured a facial image of theindividual. However, the facial image may be incomplete or inconclusivebecause the individual may be wearing clothing or other apparel, such asa hat, that obscures facial features. In such a case, the remotecomputing facility may request that a different type of biometriccapture be used and additional images or data be transmitted. In thecase described above, the user may be directed to obtain an iris imageto supplement the captured facial image. In other instances, theadditional requested data may be an audio sample of the individual'svoice.

FIG. 40 illustrates capturing an iris image for iris recognition. Thefigure illustrates the focus parameters used to analyze the image andincludes a geographical location of the individual at the time ofbiometric data capture. FIG. 40 also depicts a sample report that isdisplayed on the eyepiece.

FIG. 41 illustrates capture of multiple types of biometric data, in thisinstance, facial and iris images. The capture may be done at the sametime, or by request of the remote computing facility if a first type ofbiometric data leads to an inconclusive result.

FIG. 42 shows the electrical configuration of the multiple microphonearrays contained in the temples of the eyepiece of FIG. 39. The endfiremicrophone arrays allow for greater discrimination of signals and betterdirectionality at a greater distance. Signal processing is improved byincorporating a delay into the transmission line of the back microphone.The use of dual omni-directional microphones enables switching from anomni-directional microphone to a directional microphone. This allows forbetter direction finding for audio capture of a desired individual. FIG.43 illustrates the directionality improvements available with multiplemicrophones.

The multiple microphones may be arranged in a composite microphonearray. Instead of using one standard high quality microphone to capturean audio sample, the eyepiece temple pieces house multiple microphonesof different character. For example, this may be provided when the useris generating a biometric fingerprint of someone's voice for futurecapture and comparison. One example of multiple microphone use usesmicrophones from cut off cell phones to reproduce the exact electricaland acoustic properties of the individual's voice. This sample is storedfor future comparison in a database. If the individual's voice is latercaptured, the earlier sample is available for comparison, and will bereported to the eyepiece user, as the acoustic properties of the twosamples will match.

FIG. 44 shows the use of adaptive arrays to improve audio data capture.By modifying pre-existing algorithms for audio processing adaptivearrays can be created that allow the user to steer the directionality ofthe antenna in three dimensions. Adaptive array processing permitslocation of the source of the speech, thus tying the captured audio datato a specific individual. Array processing permits simple summing of thecardioid elements of the signal to be done either digitally or usinganalog techniques. In normal use, a user should switch the microphonebetween the omni-directional pattern and the directional array. Theprocessor allows for beamforming, array steering and adaptive arrayprocessing, to be performed on the eyepiece. In embodiments, an audiophase array may be used for audio tracking of a specific individual. Forinstance, the user may lock onto the audio signature of an individual inthe surrounding environment (such as acquired in real-time or from adatabase of sound signatures), and track the location of the individualwithout the need to maintain eye contact or the user moving their head.The location of the individual may be projected to the user through theeyepiece display. In embodiments, the tracking of an individual may alsobe provided through an embedded camera in the eyepiece, where the userwould not be required to maintain eye contact with the individual, ormove their head to follow. That is, in the case of either the audio orvisual tracking, the eyepiece may be able to track the individual withinthe local environment, without the user needing to show an physicalmotion to indicate that tracking is taking place and even as the usermoves their direction of view.

In an embodiment, the integrated camera may continuously record a videofile, and the integrated microphone may continuously record an audiofile. The integrated processor of the eyepiece may enable event taggingin long sections of the continuous audio or video recording. Forexample, a full day of passive recording may be tagged whenever anevent, conversation, encounter, or other item of interest takes place.Tagging may be accomplished through the explicit press of a button, anoise or physical tap, a hand gesture, or any other control techniquedescribed herein. A marker may be placed in the audio or video file orstored in a metadata header. In embodiments, the marker may include theGPS coordinate of the event, conversation, encounter, or other item ofinterest. In other embodiments, the marker may be time-synced with a GPSlog of the day. Other logic-based triggers can also tag the audio orvideo file such as proximity relationships to other users, devices,locations, or the like. Event tags may be active event tags that theuser triggers manually, passive event tags that occur automatically(such as through preprogramming, through an event profile managementfacility, and the like), a location-sensitive tag triggered by theuser's location, and the like. The event that triggers the event tag maybe triggered by a sound, a sight, a visual marker, received from anetwork connection, an optical trigger, an acoustic trigger, a proximitytrigger, a temporal trigger, a geo-spatial trigger, and the like. Theevent trigger may generate feedback to the user (such as an audio tone,a visual indicator, a message, and the like), store information (such asstoring a file, document, entry in a listing, an audio file, a videofile, and the like), generate an informational transmission, and thelike.

In an embodiment, the eyepiece may be used as SigInt Glasses. Using oneor more of an integrated WiFi, 3G or Bluetooth radios, the eyepiece maybe used to conspicuously and passively gather signals intelligence fordevices and individuals in the user's proximity. Signals intelligencemay be gathered automatically or may be triggered when a particulardevice ID is in proximity, when a particular audio sample is detected,when a particular geo-location has been reached, and the like.

Various embodiments of tactical glasses may include standaloneidentification or collection of biometrics to geo-locate POIs, withvisual biometrics (face, iris, walking gait) at a safe distance andpositively identify POIs with robust sparse recognition algorithms forthe face and iris. The glasses may include a hands free display forbiometric computer interface to merge print and visual biometrics on onecomprehensive display with augmented target highlighting and viewmatches and warnings without alerting the POI. The glasses may includelocation awareness, such as displaying current and average speeds plusroutes and ETA to destination and preloading or recording trouble spotsand ex-filtration routes. The glasses may include real-time networkedtracking of blue and red forces to always know where your friendly'sare, achieve visual separation range between blue and red forces, andgeo-locate the enemy and share their location in real-time. A processorassociated with the glasses may include capabilities for OCR translationand speech translation.

The tactical glasses can be used in combat to provide a graphical userinterface projected on the lens that provides users with directions andaugmented reality data on such things as team member positional data,map information of the area, SWIR/CMOS night vision, vehicular S/A forsoldiers, geo locating laser range finder for geo-locating a POI or atarget to >500 m with positional accuracy of typically less than twometers, S/A blue force range rings, Domex registration, AR field repairoverlay, and real time UAV video. In one embodiment, the laser rangefinder may be a 1.55 micron eye-safe laser range finder.

The eyepiece may utilize GPS and inertial navigation (e.g. utilizing aninertial measurement unit) as described herein, such as describedherein, to provide positional and directional accuracy. However, theeyepiece may utilize additional sensors and associated algorithms toenhance positional and directional accuracy, such as with a 3-axisdigital compass, inclinometer, accelerometer, gyroscope, and the like.For instance, a military operation may require greater positionalaccuracy then is available from GPS, and so other navigation sensors maybe utilized in combination to increase the positional accuracy of GPS.

The tactical glasses may feature enhanced resolution, such as 1280×1024pixels, and may also feature auto-focus.

In dismounted and occupied enemy engagement missions, defeating alow-intensity, low-density, asymmetrical form of warfare is incumbentupon efficient information management. The tactical glasses systemincorporates ES2 (every soldier is a sensor) capabilities throughuncooperative data recording and intuitive tactical displays for acomprehensive picture of situational awareness.

In embodiments, the tactical glasses may include one or more waveguidesbeing integrated into the frame. In some embodiments, the total internalreflection lens is attached to a pair of ballistic glasses in amonocular or binocular flip-up/flip-down arrangement. The tacticalglasses may include omni-directional ear buds for advanced hearing andprotection and a noise-cancelling boom microphone for communicationphonetically differentiated commands.

In some embodiments, the waveguides may have contrast control. Thecontrast may be controlled using any of the control techniques describedherein, such as gesture control, automatic sensor control, manualcontrol using a temple mounted controller, and the like.

The tactical glasses may include a non-slip, adjustable elastichead-strap. The tactical glasses may include clip-in corrective lenses.

In some embodiments, the total internal reflection lens is attached to adevice that is helmet-mounted, such as in FIG. 74, and may include aday/night, VIS/NIR/SWIR CMOS color camera. The device enables unimpeded“sight” of the threat as well as the soldier's own weapon with “seethrough”, flip-up electro-optic projector image display. Thehelmet-mounted device, shown in FIG. 74A, may include an IR/SWIRilluminator 7402, UV/SWIR illuminator 7404, visible to SWIR panoramiclens 7408, visible to SWIR objective lens (not shown), transparentviewing pane 7410, iris recognition objective lens 7412, laser emitter7414, laser receiver 7418, or any other sensor, processor, or technologydescribed with respect to the eyepiece described herein, such as anintegrated IMU, an eye-safe laser range finder, integrated GPS receiver,compass and inclinometer for positional accuracy, perspective controlthat changes the viewing angle of the image to match the eye position,electronic image stabilization and real-time enhancement, a library ofthreats stored onboard or remotely for access over a tactical network,and the like. A body-worn wireless computer may interface with thedevice in FIG. 74. The helmet-mounted device includes visible to SWIRprojector optics, such as RGB microprojector optics. Multi-spectral IRand UV imaging helps spot fake or altered docs. The helmet-mounteddevice may be controlled with an encrypted wireless UWB wrist or weaponfore grip controller.

In an embodiment, the transparent viewing pane 7410 can rotate through180° to project imagery onto a surface to share with others.

FIG. 74B shows a side view of the exploded device mounted to a helmet.The device may include a fully ambidextrous mount for mounting on theleft or right side of the helmet. In some embodiments, two devices maybe mounted on each of the left and right sides of the helmet to enablebinocular vision. The device or devices may snap into a standard MICH orPRO-TECH helmet mount.

Today the warfighter cannot utilize fielded data devices effectively.The tactical glasses system combines a low profile form, lightweightmaterials and fast processers to make quick and accurate decisions inthe field. The modular design of the system allows the devices to beeffectively deployed to the individual, squad or company while retainingthe ability to interoperate with any fielded computer. The tacticalglasses system incorporates real-time dissemination of data. With theonboard computer interface the operator can view, upload or compare datain real time. This provides valuable situational and environmental datacan be rapidly disseminated to all networked personnel as well ascommand posts (CPs) and tactical operations centers (TOCs).

FIGS. 75A and 75B in a front and side view, respectively, depict anexemplary embodiment of biometric and situational awareness glasses.This embodiment may include multiple field of view sensors 7502 forbiometric collection situational awareness and augmented view userinterface, fast locking GPS receiver and IMU, including 3-axis digitalcompass, gyroscope, accelerometer and inclinometer for positional anddirectional accuracy, 1.55 micron eye-safe laser range finder 7504 toassist biometric capture and targeting integrated digital video recorderstoring two Flash SD cards, real-time electronic image stabilization andreal-time image enhancement, library of threats stored in onboardmini-SD card or remotely loaded over a tactical network, flip-upphotochromic lenses 7508, noise-cancelling flexible boom mike 7510 and3-axis detachable stereo ear buds plus augmented hearing and protectionsystem 7512. For example, the multiple field of view sensors 7502 mayenable a 100°×40° FOV, which may be panoramic SXGA. For example, thesensors may be a VGA sensor, SXGA sensor, and a VGA sensor thatgenerates a panoramic SXGA view with stitched 100°×40° FOV on a displayof the glasses. The displays may be translucent with perspective controlthat changes the viewing angle of the image to match the eye position.This embodiment may also include SWIR detection to let wearers see 1064nm and 1550 nm laser designators, invisible to the enemy and may featureultra-low power 256-bit AES Encrypted connection between glasses,tactical radios and computers, instant 2× zoom, auto face tracking, faceand iris recording, and recognition and GPS geo-location with a 1 mauto-recognition range. This embodiment may include a power supply, suchas a 24 hour duration 4-AA alkaline, lithium and rechargeable batterybox with its computer and memory expansion slots with a water- anddust-proof cord. In an embodiment, the glasses include a curvedholographic wave guide.

In embodiments, the eyepiece may be able to sense lasers such as used inbattlefield targeting. For instance, sensors in the eyepiece may be ableto detect laser light in typical military-use laser transmission bands,such as 1064 nm, 1550 nm, and the like. In this way, the eyepiece may beable to detect whether their position is being targeted, if anotherlocation is being targeted, the location of a spotter using the laser asa targeting aid, and the like. Further, since the eyepiece may be ableto sense laser light, such as directly or reflected, the soldier may notonly detect enemy laser sources that have been directed or reflected totheir position, but may supply the laser source themselves in order tolocate optical surfaces (e.g. binoculars) in the battlefield scene. Forexample, the soldier scans the field with a laser and watches with theeyepiece for a reflected return of the laser as a possible location ofan enemy viewing though binoculars. In embodiments, the eyepiece maycontinuously scan the surrounding environment for laser light, andprovide feedback and/or action as a result of a detection, such as anaudible alarm to the soldier, a location indicted through a visualindicator on the eyepiece display, and the like.

In some embodiments, a Pocket Camera may video record and captures stillpictures, allowing the operator to record environmental data foranalysis with a mobile, lightweight, rugged biometric device sized to bestored in a pocket. An embodiment may be 2.25″×3.5″×0.375″ and capableof face capture at 10 feet, iris capture at 3 feet, recording voice,pocket litter, walking gait, and other identifying visible marks andenvironmental data in EFTS and EBTS compliant formatting compatible withany Iris/Face algorithm. The device is designed to pre-qualify andcapture EFTS/EBTS/NIST/ISO/ITL 1-2007 compliant salient images to bematched and filed by any biometric matching software or user interface.The device may include a high definition video chip, 1 GHz processorwith 533 Mhz DSP, GPS chip, active illumination and pre-qualificationalgorithms. In some embodiments, the Pocket Bio Cam may not incorporatea biometric watch list so it can be used at all echelons and/or forconstabulary leave-behind operations. Data may be automaticallygeo-located and date/time stamped. In some embodiments, the device mayoperate Linux SE OS, meet MIL-STD-810 environmental standards, and bewaterproof to 3 ft depth.

In an embodiment, a device for collection of fingerprints may be knownas a bio-print device. The bio-print apparatus comprises a clear platenwith two beveled edges. The platen is illuminated by a bank of LEDs andone or more cameras. Multiple cameras are used and are closely disposedand directed to the beveled edge of the platen. A finger or palm isdisposed over the platen and pressed against an upper surface of theplaten, where the cameras capture the ridge pattern. The image isrecorded using frustrated total internal reflection (FTIR). In FTIR,light escapes the platen across the air gap created by the ridges andvalleys of the fingers or palm pressed against the platen.

Other embodiments are also possible. In one embodiment, multiple camerasare place in inverted ‘V’s of a saw tooth pattern. In anotherembodiment, a rectangle is formed and uses light direct through one sideand an array of cameras capture the images produced. The light entersthe rectangle through the side of the rectangle, while the cameras aredirectly beneath the rectangle, enabling the cameras to capture theridges and valleys illuminated by the light passing through therectangle.

After the images are captured, software is used to stitch the imagesfrom the multiple cameras together. A custom FPGA may be used for thedigital image processing.

Once captured and processed, the images may be streamed to a remotedisplay, such as a smart phone, computer, handheld device, or eyepiece,or other device.

The above description provides an overview of the operation of themethods and apparatus of the disclosure. Additional description anddiscussion of these and other embodiments is provided below.

FIG. 45 illustrates the construction and layout of an optics basedfinger and palm print system according to an embodiment. The opticalarray consists of approximately 60 wafer scale cameras 4502. The opticsbased system uses sequential perimeter illumination 4503, 4504 for highresolution imaging of the whorls and pores that comprise a finger orpalm print. This configuration provides a low profile, lightweight, andextremely rugged configuration. Durability is enhanced with a scratchproof, transparent platen.

The mosaic print sensor uses a frustrated total internal reflection(FTIR) optical faceplate provides images to an array of wafer scalecameras mounted on a PCB like substrate 4505. The sensor may be scaledto any flat width and length with a depth of approximately ½″. Size mayvary from a plate small enough to capture just one finger roll print, upto a plate large enough to capture prints of both hands simultaneously.

The mosaic print sensor allows an operator to capture prints and comparethe collected data against an on-board database. Data may also beuploaded and downloaded wirelessly. The unit may operate as a standaloneunit or may be integrated with any biometric system.

In operation the mosaic print sensor offers high reliability in harshenvironments with excessive sunlight. To provide this capability,multiple wafer scale optical sensors are digitally stitched togetherusing pixel subtraction. The resulting images are engineered to be over500 dots per inch (dpi). Power is supplied by a battery or byparasitically drawing power from other sources using a USB protocol.Formatting is EFTS, EBTS NIST, ISO, and ITL 1-2007 compliant.

FIG. 46 illustrates the traditional optical approach used by othersensors. This approach is also based on FTIR. In the figure, the fringescontact the prism and scatter the light. The fringes on the finger beingprinted show as dark lines, while the valleys of the fingerprint show asbright lines.

FIG. 47 illustrates the approach used by the mosaic sensor 4700. Themosaic sensor also uses FTIR. However, the plate is illuminated from theside and the internal reflections are contained within the plate of thesensor. The fringes contact the prism and scatter the light, allowingthe camera to capture the scattered light. The fringes on the fingershow as bright lines, whiles the valleys show as dark lines.

FIG. 48 depicts the layout of the mosaic sensor 4800. The LED array isarranged around the perimeter of the plate. Underneath the plate are thecameras used to capture the fingerprint image. The image is captured onthis bottom plate, known as the capture plane. The capture plane isparallel to the sensor plane, where the fingers are placed. Thethickness of the plate, the number of the cameras, and the number of theLEDs may vary, depending on the size of the active capturing area of theplate. The thickness of the plate may be reduced by adding mirrors thatfold the optical path of the camera, reducing the thickness needed. Eachcamera should cover one inch of space with some pixels overlappingbetween the cameras. This allows the mosaic sensor to achieve 500 ppi.The cameras may have a field of view of 60 degrees; however, there maybe significant distortion in the image.

FIG. 49 shows an embodiment 4900 of a camera field of view and theinteraction of the multiple cameras used in the mosaic sensor. Eachcamera covers a small capturing area. This area depends on the camerafield of view and the distance between the camera and the top surface ofthe plate. α is one half of the camera's horizontal field of view and βis one half of the camera's vertical field of view.

The mosaic sensor may be incorporated into a bio-phone and tacticalcomputer as illustrated in FIG. 50. The bio-phone and tactical computeruses a completed mobile computer architecture that incorporates dualcore processors, DSP, 3-D graphics accelerator, 3G-4G Wi-Lan (inaccordance with 802.11a/b/g/n), Bluetooth 3.0, and a GPS receiver. Thebio-phone and tactical computer delivers power equivalent to a standardlaptop in a phone size package.

FIG. 50 illustrates the components of the bio-phone and tacticalcomputer. The bio-phone and tactical computer assembly, 5000 provides adisplay screen 5001, speaker 5002 and keyboard 5003 contained withincase 5004. These elements are visible on the front of the bio-phone andtactical computer assembly 5000. On the rear of the assembly 3800 arelocated a camera for iris imaging 5005, a camera for facial imaging andvideo recording 5006 and a bio-print fingerprint sensor 5009.

To provide secure communications and data transmission, the deviceincorporates selectable 256-bit AES encryption with COTS sensors andsoftware for biometric pre-qualification for POI acquisition. Thissoftware is matched and filed by any approved biometric matchingsoftware for sending and receiving secure “perishable” voice, video, anddata communications. In addition, the bio-phone supports Windows Mobile,Linux, and Android operating systems.

The bio-phone is a 3G-4G enabled hand-held device for reach back to webportals and biometric enabled watch list BEWL) databases. Thesedatabases allow for in-field comparison of captured biometric images anddata. The device is designed to fit into a standard LBV or pocket. Inembodiments, the biometrics phone and tactical computer may use a mobilecomputer architecture featuring dual core processors, DSP, 3-D graphicsaccelerator, 3G-4G, Wi-LAN (802.11a/b/g/n), Bluetooth 3.0, enabled forsecure and civilian networks, GPS Receiver, WVGA sun-sight readablecapacitance touch-screen display, capable of outputting stereoscopic 3Dvideo, tactile backlit QWERTY keyboard, on-board storage, supportingmultiple operating systems, and the like, that delivers laptop power ina light weight design.

The bio-phone can search, collect, enroll, and verify multiple types ofbiometric data, including face, iris, two-finger fingerprint, as well asbiographic data. The device also records video, voice, gait, identifyingmarks, and pocket litter. Pocket litter includes a variety of smallitems normally carried in a pocket, wallet, or purse and may includesuch items as spare change, identification, passports, charge cards, andthe like. FIG. 52 shows a typical collection of this type ofinformation. Depicted in FIG. 52 are examples of a collection of pocketlitter 5200. The types of items that may be included are personaldocuments and pictures 5201, books 5202, notebooks and paper, 5203, anddocuments, such as a passport 5204.

The biometrics phone and tactical computer may include a camera, such asa high definition still and video camera, capable of biometric datataking and video conferencing. In embodiments, the eyepiece camera andvideoconference capabilities, as described herein, may be used inconjunction with the biometrics phone and tactical computer. Forinstance, a camera integrated into the eyepiece may capture images andcommunicate the images to the biometrics phone and tactical computer,and vice a versa. Data may be exchanged between the eyepiece andbiometrics phone, network connectivity may be established by either, andshared, and the like. In addition, the biometric phone and tacticalcomputer may be housed in a rugged, fully militarized construction,tolerant to a militarized temperature range, waterproof (such as to adepth of 5 m), and the like.

FIG. 51 illustrates an embodiment 5100 of the use of the bio-phone tocapture latent fingerprints and palm prints. Fingerprints and palmprints are captured at 1000 dpi with active illumination from anultraviolet diode with scale overlay. Both fingerprint and palm prints5100 may be captured using the bio-phone.

Data collected by the bio-phone is automatically geo-located and dateand time stamped using the GPS capability. Data may be uploaded ordownloaded and compared against onboard or networked databases. Thisdata transfer is facilitated by the 3G-4G, Wi-Lan, and Bluetoothcapabilities of the device. Data entry may be done with the QWERTYkeyboard, or other methods that may be provided, such as stylus or touchscreen, or the like. Biometric data is filed after collection using themost salient image. Manual entry allows for partial data capture. FIG.53 illustrates the interplay 5300 between the digital dossier images andthe biometric watch list held at a database. The biometric watch list isused for comparing data captured in the field with previously captureddata

Formatting may use EFTS, EBTS NIST, ISO, and ITL 1-2007 formats toprovide compatibility with a range and variety of databases forbiometric data.

The specifications for the bio-phone and tactical computer are givenbelow:

-   -   Operating Temperature: −22° C. to +70° C.    -   Connectivity I/O: 3G, 4G, WLAN a/b/g/n, Bluetooth 3.0, GPS, FM    -   Connectivity Output: USB 2.0, HDMI, Ethernet    -   Physical Dimensions: 6.875″ (H)×4.875″ (W)×1.2″ (T)    -   Weight: 1.75 lbs.    -   Processor: Dual Core—1 GHz Processors, 600 MHz DSP, and 30M        Polygon/sec 3-D Graphics Accelerator    -   Display: 3,8″ WVGA (800×480) Sunlight Readable, Transreflective,        Capacitive Touch Screen, Scalable display output for connection        to 3×1080 p Hi-Def screens simultaneously.    -   Operating System Windows Mobile, Linux, SE, Android    -   Storage: 128 GB solid-state drive    -   Additional Storage Dual SD Card slots for additional 128 GB        storage.    -   Memory: 4 GB RAM    -   Camera: 3 Hi-Def Still and Video Cameras: Face, Iris, and        Conference        -   (User's Face)    -   3D Support: Capable of outputting stereoscopic 3D video.        -   Camera Sensor Support: Sensor dynamic range extension,            Adaptive defect pixel correction, advanced sharpness            enhancement, Geometric distortion correction, advanced color            management, HW based face detection, Video stabilization    -   Biometrics: On-board optical, 2 fingerprint sensor, Face, DOMEX,        and Iris cameras.    -   Sensors: Can accommodate the addition of accelerometer, compass,        ambient light, proximity, barometric, and temperature sensors,        depending on requirements.    -   Battery: <8 hrs, 1400 Mah, rechargeable Li-ion, hot swap battery        pack.    -   Power: Various power options for continuous operation.    -   Software Features Face/gesture detection, noise filtering, pixel        correction.

Powerful display processor with multi-overlay, rotation, and resizingcapabilities.

-   -   Audio: On board microphone, speakers, and audio/video inputs.    -   Keyboard: Full tactile QWERTY keyboard with adjustable        backlight.

Additional devices and kits may also incorporate the mosaic sensors andmay operate in conjunction with the bio-phone and tactical computer toprovide a complete field solution for collection biometric data.

One such device is the pocket bio-kit, illustrated in FIG. 54. Thecomponents of the pocket bio-kit 5400 include a GPS antenna 5401, abio-print sensor 5402, keyboard 5404, all contained in case 5403. Thespecifications of the bio-kit are given below:

-   -   Size: 6″×3″×1.5″    -   Weight: 2 lbs. total    -   Processor and Memory: 1 GHz OMAP processor        -   650 MHz core        -   3-D accelerator handling up to 18 million polygons/sec        -   64 KB L2 cache        -   166 MHz at 32 bit FSB        -   1 GB embedded PoP memory expandable with up to 4 GB NAND        -   64 GB solid state hard drive    -   Display: 75 mm×50 mm, 640×480 (VGA) daylight readable LCD,        anti-glare, anti-reflective, anti-scratch screen treatment    -   Interface: USB 2.0        -   10/100/1000 Ethernet    -   Power: Battery operation: approximately 8 hours of continuous        enrollments at roughly 5 minutes per enrollment.    -   Embedded Capabilities: mosaic sensor optical fingerprint reader        -   Digital iris camera with active IR illumination        -   Digital face and DOMEX camera (visible) with flash        -   Fast lock GPS

The features of the bio-phone and tactical computer may also be providedin a bio-kit that provides for a biometric data collection system thatfolds into a rugged and compact case. Data is collected in biometricstandard image and data formats that can be cross-referenced for nearreal-time data communication with Department of Defense BiometricAuthoritative Databases.

The pocket bio-kit shown in FIG. 55 can capture latent fingerprints andpalm prints at 1,000 dpi with active illumination from an ultravioletdiode with scale overlay. The bio-kit holds 32 GB memory storage cardsthat are capable of interoperation with combat radios or computers forupload and download of data in real-time field conditions. Power isprovided by lithium ion batteries. Components of the bio-kit assembly5500 include a GPS antenna 5501, a bio-print sensor 5502, and a case5503 with a base bottom 5505.

Biometric data collect is geo-located for monitoring and trackingindividual movement. Finger and palm prints, iris images, face images,latent fingerprints, and video may be collected and enrolled in adatabase using the bio-kit. Algorithms for finger and palm prints, irisimages, and face images facilitate these types of data collection. Toaid in capturing iris images and latent fingerprint imagessimultaneously, the bio-kit has IR and UV diodes that activelyilluminate an iris or latent fingerprint. In addition, the pocketbio-kit is also fully EFTS/EBTS compliant, including ITL 1-2007 and WSQ.The bio-kit meets MIL-STD-810 for operation in environmental extremesand uses a Linux operating system.

For capturing images, the bio-kit uses a high dynamic range camera withwave front coding for maximum depth of field, ensuring detail in latentfingerprints and iris images is captured. Once captured, real-time imageenhancement software and image stabilization act to improve readabilityand provide superior visual discrimination.

The bio-kit is also capable of recording video and stores full-motion(30 fps) color video in an onboard “camcorder on chip.”

The eyepiece 100 may interface with the mobile folding biometricsenrollment kit (aka bio-kit) 5500, a biometric data collection systemthat folds into a compact rugged case, such that unfolds into a miniworkstation for fingerprints, iris and facial recognition, latentfingerprint, and the like biometric data as described herein. As is thecase for the other mobile biometrics devices, the mobile foldingbiometrics enrollment kit 5500 may be used as a stand-alone device or inassociation with the eyepiece 100, as described herein. In anembodiment, the mobile folding biometrics enrollment kit may fold up toa small size such as 6″×3″×1.5″ with weight such as 2 pounds. It maycontain a processor, digital signal processor, 3D accelerator, fastsyndrome-based hash (FSB) functions, solid state memory (e.g.package-on-package (PoP)), hard drive, display (e.g. 75 mm×50 mm,640×480 (VGA) daylight-readable LCD anti-glare, anti-reflective, antiscratch screen), USB, Ethernet, embedded battery, mosaic opticalfingerprint reader, digital iris camera (such as with active IRillumination), digital face and DOMEX camera with flash, fast lock GPS,and the like. Data may be collected in biometric standard image and dataformats that may be cross-referenced for a near real-time datacommunication with the DoD biometric authoritative databases. The devicemay be capable of collecting biometric data and geo-location of personsof interest for monitoring and tracking, wireless data upload/downloadusing combat radio or computer with standard networking interface, andthe like.

In addition to the bio-kit, the mosaic sensor may be incorporated into awrist mounted fingerprint, palm print, geo-location, and POI enrollmentdevice, shown in FIG. 56. The eyepiece 100 may interface with thebiometric device 5600, a biometric data collection system that straps ona soldier's wrist or arm and folds open for fingerprints, irisrecognition, computer, and the like biometric data as described herein.The device may have an integrated computer, keyboard, sunlight-readabledisplay, biometric sensitive platen, and the like, so operators mayrapidly and remotely store or compare data for collection andidentification purposes. For instance, the arm strap biometric sensitiveplaten may be used to scan a palm, fingerprints, and the like. Thedevice may provide geo-location tags for person of interest andcollected data with time, date, location, and the like. As is the casefor the other mobile biometrics devices, the biometric device 5600 maybe used as a stand-alone device or in association with the eyepiece 100,as described herein. In an embodiment, the biometric device may be smalland light to allow it to be comfortably worn on a soldier's arm, such aswith dimensions 5″×2.5″ for the active fingerprint and palm printsensor, and a weight of 16 ounces. There may be algorithms forfingerprint and palm capture. The device may include a processor,digital signal processor, a transceiver, a Qwerty key board, largeweather-resistant pressure driven print sensor, sunlight readabletransflective QVGA color backlit LCD display, internal power source, andthe like.

In one embodiment, the wrist mounted assembly 5600 includes thefollowing elements in case 5601: straps 5602, setting and on/off buttons5603, protective cover for sensor 5604, pressure-driven sensor 4405, anda keyboard and LCD screen 5606.

The fingerprint, palm print, geo-location, and POI enrollments deviceincludes an integrated computer, QWERTY keyboard, and display. Thedisplay is designed to allow easy operation in strong sunlight and usesan LCD screen or LED indicator to alert the operator of successfulfingerprint and palm print capture. The display uses transflective QVGAcolor, with a backlit LCD screen to improve readability. The device islightweight and compact, weighing 16 oz. and measuring 5″×2.5″ at themosaic sensor. This compact size and weight allows the device to slipinto an LBV pocket or be strapped to a user's forearm, as shown in FIG.56. As with other devices incorporating the mosaic sensor, all POIs aretagged with geo-location information at the time of capture.

The size of the sensor screen allows 10 fingers, palm, four-finger slap,and finger tip capture. The sensor incorporates a large pressure drivenprint sensor for rapid enrollment in any weather conditions as specifiedin MIL-STD-810, at a rate of 500 dpi. Software algorithms support bothfingerprint and palm print capture modes and uses a Linux operatingsystem for device management. Capture is rapid, due to the 720 MHzprocessor with 533 MHZ DSP. This processing capability deliverscorrectly formatted, salient images to any existing approved systemsoftware. In addition, the device is also fully EFTS/EBTS compliant,including ITL 1-2007 and WSQ.

As with other mosaic sensor devices, communication in wireless mode ispossible using a removable UWB wireless 256-bit AES transceiver. Thisalso provides secure upload and download to and from biometric databasesstored off the device.

Power is supplied using lithium polymer or AA alkaline batteries.

The wrist-mounted device described above may also be used in conjunctionwith other devices, including augmented reality eyepieces with data andvideo display, shown in FIG. 57. The assembly 5700 includes thefollowing components: an eyepiece 5702, and a bio-print sensor device5700. The augmented reality eyepiece provides redundant, binocular,stereo sensors and display and provides the ability to see in a varietyof lighting conditions, from glaring sun at midday, to the extremely lowlight levels found at night Operation of the eyepiece is simple with arotary switch located on the temple of the eyepiece a user can accessdata from a forearm computer or sensor, or a laptop device. The eyepiecealso provides omni-directional earbuds for hearing protection andimproved hearing. A noise cancelling boom microphone may also beintegrated into the eyepiece to provide better communication ofphonetically differentiated commands.

The eyepiece is capable of communicating wirelessly with the bio-phonesensor and forearm mounted devices using a 256-bit AES encrypted UWB.This also allows the device to communicate with a laptop or combatradio, as well as network to CPs, TOCs, and biometric databases. Theeyepiece is ABIS, EBTS, EFTS, and JPEG 2000 compatible.

Similar to other mosaic sensor devices described above, the eyepieceuses a networked GPS to provide highly accurate geo-location of POIs, aswell as a RF filter array.

In operation the low profile forearm mounted computer and tacticaldisplay integrate face, iris, fingerprint, palm print, and fingertipcollection and identification. The device also records video, voice,gait, and other distinguishing characteristics. Facial and iris trackingis automatic, allowing the device to assist in recognizingnon-cooperative POIs. With the transparent display provided by theeyepiece, the operator may also view sensor imagery, moving maps,superimposed applications with navigation, targeting, position or otherinformation from sensors, UAVs, and the like, and data as well as theindividual whose biometric data is being captured or other targets/POIs.

FIG. 58 illustrates a further embodiment of the fingerprint, palm print,geo-location, and POI enrollment device. The device is 16 oz and uses a5″×2.5″ active fingerprint and palm print capacitance sensor. The sensoris capable of enrolling 10 fingers, a palm, 4 finger slap, and fingertip prints at 500 dpi. A 0.6-1 GHz processor with 430 MHz DSP providesrapid enrollment and data capture. The device is ABIS, EBTS, EFTS, andJPEG 2000 compatible and features networked GPS for highly accuratelocation of persons of interest. In addition, the device communicateswirelessly over a 256-bit AES encrypted UWB, laptop, or combat radio.Database information may also be stored on the device, allowing in thefield comparison without uploading information. This onboard data mayalso be shared wirelessly with other devices, such as a laptop or combatradio.

A further embodiment of the wrist mounted bio-print sensor assembly 5800includes the following elements: a bio-print sensor 5801, wrist strap5802, keyboard 5803, and combat radio connector interface 5804.

Data may be stored on the forearm device since the device can utilizeMil-con data storage caps for increased storage capacity. Data entry isperformed on the QWERTY keyboard and may be done wearing gloves.

The display is a transflective QVGA, color, backlit LCD display designedto be readable in sunlight. In addition to operation in strong sunlight,the device may be operated in a wide range of environments, as thedevice meets the requirements of MIL-STD-810 operation in environmentalextremes.

The mosaic sensor described above may also be incorporated into amobile, folding biometric enrollment kit, as shown in FIG. 59. Themobile folding biometric enrollment kit 5900 folds into itself and issized to fit into a tactical vest pocket, having dimensions of 8×12×4inches when unfolded.

FIG. 60 illustrates an embodiment 6000 of how the eyepiece and forearmmounted device may interface to provide a complete system for biometricdata collection.

FIG. 61 provides a system diagram 6100 for a mobile folding biometricenrollment kit.

In operation the mobile folding biometric enrollment kit allows a userto search, collect, identify, verify, and enroll face, iris, palm print,fingertip, and biographic data for a subject and may also record voicesamples, pocket litter, and other visible identifying marks. Oncecollected, the data is automatically geo-located, date, and timestamped. Collected data may be searched and compared against onboard andnetworked databases. For communicating with databases not onboard thedevice, wireless data up/download using combat radio or laptop computerwith standard networking interface is provided. Formatting is compliantwith EFTS, EBTS, NIST, ISO, and ITL 1-2007. Prequalified images may besent directly to matching software as the device may use any matchingand enrollment software.

The devices and systems incorporating described above provide acomprehensive solution for mobile biometric data collection,identification, and situational awareness. The devices are capable ofcollecting fingerprints, palm prints, fingertips, faces, irises, voice,and video data for recognition of uncooperative persons of interest(POI). Video is captured using high speed video to enable capture inunstable situations, such as from a moving video. Captured informationmay be readily shared and additional data entered via the keyboard. Inaddition, all data is tagged with date, time, and geo-location. Thisfacilitates rapid dissemination of information necessary for situationalawareness in potentially volatile environments. Additional datacollection is possible with more personnel equipped with the devices,thus, demonstrating the idea that “every soldier is a sensor.” Sharingis facilitated by integration of biometric devices with combat radiosand battlefield computers.

In embodiments, the eyepiece may utilize flexible thin-film sensors,such as integrated into the eyepiece itself, into an external devicethat the eyepiece interfaces with, and the like. A thin film sensor maycomprise a thin multi-layer electromechanical arrangement that producesan electrical signal when subjected to a sudden contact force or tocontinuously varying forces. Typical applications of electromechanicalthin film sensors employ both on-off electrical switch sensing and thetime-resolved sensing of forces. Thin-film sensors may include switches,force gauges, and the like, where thin film sensors may rely upon theeffects of sudden electrical contact (switching), the gradual change ofelectrical resistance under the action of force, the gradual release ofelectrical charges under the action of stress forces, the generation ofa gradual electromotive force across a conductor when, moving in amagnetic field, and the like. For example, flexible thin-film sensorsmay be utilized in force-pressure sensors with microscopic forcesensitive pixels for two-dimensional force array sensors. This may beuseful for touch screens for computers, smart-phones, notebooks,MP-3-like devices, especially those with military applications; screensfor controlling anything under computer control including unmannedaerial vehicles (UAV), drones, mobile robots, exoskeleton-based devices;and the like. Thin-film sensors may be useful in security applications,such as in remote or local sensors for detecting intrusion, opening orclosing of devices, doors, windows, equipment, and the like. Thin-filmsensors may be useful for trip wire detection, such as with electronicsand radio used in silent, remote trip-wire detectors. Thin-film sensorsmay be used in open-close detections, such as force sensors fordetecting strain-stress in vehicle compartments, ship hulls, aircraftpanels, and the like. Thin-film sensors may be useful as biometricsensors, such as in fingerprinting, palm-printing, finger tip printing,and the like. Thin-film sensors may be useful leak detection, such asdetecting leaking tanks, storage facilities, and the like. Thin-filmsensors may be useful in medical sensors, such as in detecting liquid orblood external to a body, and the like. These sensor applications aremeant to be illustrative of the many applications thin-film sensors maybe employed in association with control and monitoring of externaldevices through the eyepiece, and are not meant to be limiting in anyway.

FIG. 62 illustrates an embodiment 6200 of a thin-film finger and palmprint collection device. The device can record four fingerprint slapsand rolls, palm prints, and fingerprints to the NIST standard. Superiorquality finger print images can be captured with either wet or dryhands. The device is reduced in weight and power consumption compared toother large sensors. In addition, the sensor is self-contained and ishot swappable. The configuration of the sensor may be varied to suit avariety of needs, and the sensor may be manufactured in various shapesand dimensions.

FIG. 63 depicts an embodiment 6300 of a finger, palm, and enrollmentdata collection device. This device records fingertip, roll, slap, andpalm prints. A built in QWERTY keyboard allows entry of writtenenrollment data. As with the devices described above, all data is taggedwith date, time, and geo-location of collection. A built in databaseprovides on board matching of potential POIs against the built indatabase. Matching may also be performed with other databases over abattlefield network. This device can be integrated with the opticalbiometric collection eyepiece described above to support face and irisrecognition.

The specifications for the finger, palm, and enrollment device are givenbelow:

Weight & Size: 16 oz. forearm straps or inserts into LBV pocket

-   -   5″×2.5″ finger/palm print sensor    -   5.75″×2.75″ QWERTY keyboard    -   3.5″×2.25″ LCD display    -   One-handed operation

Environmental: Sensor operates in all weather conditions, −20° C. to+70° C.

-   -   Waterproofing: 1 m for 4 hours, operates without degradation

Biometric Collection: fingerprint and palm print collection,identification

-   -   Keyboard & LCD display for enrollment of POIs    -   Retains >30,000 full template portfolios (2 iris, 10        fingerprint, facial image, 35 fields of biographic information)        for on board matching of POIs.    -   Tags all collected biometric data with time, date, and location    -   Pressure capacitance finger/palm print sensor    -   30 fps high contrast bitmap image    -   1000 dpi

Wireless: fully interoperable with combat radios, hand held or lap topcomputers and 256-bit AES encryption

Battery: dual 2000 mAh lithium polymer batteries

-   -   >12 hours, quick change battery in <15 seconds

Processing & Memory: 256 MB flash and 128 MB SDRA supports 3 SD cards upto 32 GB each

-   -   600-1 GHZ ARM Cortex A8 processor    -   1 GB RAM

FIGS. 64-66 depict use of the devices incorporating a sensor forcollecting biometric data. FIG. 64 shows an embodiment 6400 of thecapture of a two-stage palm print. FIG. 65 shows collection 6500 using afingertip tap. FIG. 66 demonstrates an embodiment 6600 of a slap androll print being collected.

The discussion above pertains to methods of gathering biometric data,such as fingerprints or palm prints using a platen or touch screen, asshown in FIGS. 66 and 62-66. This disclosure also includes methods andsystems for touchless or contactless fingerprinting using polarizedlight. In one embodiment, fingerprints may be taken by persons using apolarized light source and retrieving images of the fingerprints usingreflected polarized light in two planes. In another embodiment,fingerprints may be taken by persons using a light source and retrievingimages of the fingerprints using multispectral processing, e.g., usingtwo imagers at two different locations with different inputs. Thedifferent inputs may be caused by using different filters or differentsensors/imagers. Applications of this technology may include biometricchecks of unknown persons or subjects in which the safety of the personsdoing the checking may be at issue.

In this method, an unknown person or subject may approach a checkpoint,for example, to be allowed further travel to his or her destination. Asdepicted in the system 6700 shown in FIG. 67, the person P and anappropriate body part, such as a hand, a palm P, or other part, areilluminated by a source of polarized light 6701. As is well known tothose with skill in optical arts, the source of polarized light maysimply be a lamp or other source of illumination with a polarizingfilter to emit light that is polarized in one plane. The light travelsto the person in an area which has been specified for non-contactfingerprinting, so that the polarized light impinges on the fingers orother body part of the person P. The incident polarized light is thenreflected from the fingers or other body part and passes in alldirections from the person. Two imagers or cameras 6704 receive thereflected light after the light has passed through optical elements suchas a lens 6702 and a polarizing filter 6703. The cameras or imagers maybe mounted on the augmented reality glasses, as discussed above withrespect to FIG. 8F.

The light then passes from palm or finger or fingers of the person ofinterest to two different polarizing filters 6704 a, 6704 b and then tothe imagers or cameras 6705. Light which has passed through thepolarizing filters may have a 90° orientation difference (horizontal andvertical) or other orientation difference, such as 30°, 45°, 60° or120°. The cameras may be digital cameras with appropriate digitalimaging sensors to convert the incident light into appropriate signals.The signals are then processed by appropriate processing circuitry 6706,such as digital signal processors. The signals may then be combined in aconventional manner, such as by a digital microprocessor with memory6707. The digital processor with appropriate memory is programmed toproduce data suitable for an image of a palm, fingerprint, or otherimage as desired. The digital data from the imagers may then be combinedin this process, for example, using the techniques of U.S. Pat. No.6,249,616 and others. As noted above in the present disclosure, thecombined “image” may then be checked against a database to determine anidentity of the person. The augmented reality glasses may include such adatabase in the memory, or may refer the signals data elsewhere 6708 forcomparison and checking.

A process for taking contactless fingerprints, palm prints or otherbiometric prints is disclosed in the flowchart of FIG. 68. In oneembodiment, a polarized light source is provided 6801. In a second step6802, the person of interest and the selected body part is positionedfor illumination by the light. In another embodiment, it may be possibleto use incident white light rather than using a polarized light source.When the image is ready to be taken, light is reflected 6803 from theperson to two cameras or imagers. A polarizing filter is placed in frontof each of the two cameras, so that the light received by the cameras ispolarized 6804 in two different planes, such as in a horizontal andvertical plane. Each camera then detects 6805 the polarized light. Thecameras or other sensors then convert the incidence of light intosignals or data 6806 suitable for preparation of images. Finally, theimages are then combined 6807 to form a very distinct, reliable print.The result is an image of very high quality that may be compared todigital databases to identify the person and to detect persons ofinterest.

It should be understood that while digital cameras are used in thiscontactless system, other imagers may be used, such as active pixelimagers, CMOS imagers, imagers that image in multiple wavelengths, CCDcameras, photo detector arrays, TFT imagers, and so forth. It shouldalso be understood that while polarized light has been used to createtwo different images, other variations in the reflected light may alsobe used. For example, rather than using polarized light, white light maybe used and then different filters applied to the imagers, such as aBayer filter, a CYGM filter, or an RGBE filter. In other embodiments, itmay be possible to dispense with a source of polarized light and insteaduse natural or white light rather than a source of polarized light.

The use of touchless or contactless fingerprinting has been underdevelopment for some time, as evidenced by earlier systems. For example,U.S. Pat. Appl. 2002/0106115 used polarized light in a non-contactsystem, but required a metallic coating on the fingers of the personbeing fingerprinted. Later systems, such as those described in U.S. Pat.No. 7,651,594 and U.S. Pat. Appl. Publ. 2008/0219522, required contactwith a platen or other surface. The contactless system described hereindoes not require contact at the time of imaging, nor does it requireprior contact, e.g., placing a coating or a reflective coating on thebody part of interest. Of course, the positions of the imagers orcameras with respect to each other should be known for easierprocessing.

In use, the contactless fingerprint system may be employed at acheckpoint, such as a compound entrance, a building entrance, a roadsidecheckpoint or other convenient location. Such a location may be onewhere it is desirable to admit some persons and to refuse entrance oreven detain other persons of interest. In practice, the system may makeuse of an external light source, such as a lamp, if polarized light isused. The cameras or other imagers used for the contactless imaging maybe mounted on opposite sides of one set of augmented reality glasses(for one person). For example, a two-camera version is shown in FIG. 8F,with two cameras 870 mounted on frame 864. In this embodiment, thesoftware for at least processing the image may be contained within amemory of the augmented reality glasses. Alternatively, the digital datafrom the cameras/imagers may be routed to a nearby datacenter forappropriate processing. This processing may include combining thedigital data to form an image of the print. The processing may alsoinclude checking a database of known persons to determine whether thesubject is of interest.

Alternatively, one camera on each of two persons may be used, as seen inthe camera 858 in FIG. 8F. In this configuration, the two persons wouldbe relatively near so that their respective images would be suitablysimilar for combining by the appropriate software. For example, the twocameras 6705 in FIG. 67 may be mounted on two different pairs ofaugmented reality glasses, such as on two soldiers manning a checkpoint.Alternatively, the cameras may be mounted on a wall or on stationaryparts of the checkpoint itself. The two images may then be combined by aremote processor with memory 6707, such as a computer system at thebuilding checkpoint.

As discussed above, persons using the augmented reality glasses may bein constant contact with each other through at least one of manywireless technologies, especially if they are both on duty at acheckpoint. Accordingly, the data from the single cameras or from thetwo-camera version may be sent to a data center or other command postfor the appropriate processing, followed by checking the database for amatch of the palm print, fingerprint, iris print, and so forth. The datacenter may be conveniently located near the checkpoint. With theavailability of modern computers and storage, the cost of providingmultiple datacenters and wirelessly updating the software will not be amajor cost consideration in such systems.

The touchless or contactless biometric data gathering discussed abovemay be controlled in several ways, such as the control techniquesdiscussed else in this disclosure. For example, in one embodiment, auser may initiate a data-gathering session by pressing a touch pad onthe glasses, or by giving a voice command. In another embodiment, theuser may initiate a session by a hand movement or gesture or using anyof the control techniques described herein. Any of these techniques maybring up a menu, from which the user may select an option, such as“begin data gathering session,” “terminate data-gathering session,” or“continue session.” If a data-gathering session is selected, thecomputer-controlled menu may then offer menu choices for number ofcameras, which cameras, and so forth, much as a user selects a printer.There may also be modes, such as a polarized light mode, a color filtermode, and so forth. After each selection, the system may complete a taskor offer another choice, as appropriate. User intervention may also berequired, such as turning on a source of polarized light or other lightsource, applying filters or polarizers, and so forth.

After fingerprints, palm prints, iris images or other desired data hasbeen acquired, the menu may then offer selections as to which databaseto use for comparison, which device(s) to use for storage, etc. Thetouchless or contactless biometric data gathering system may becontrolled by any of the methods described herein.

While the system and sensors have obvious uses in identifying potentialpersons of interest, there are positive battlefield uses as well. Thefingerprint sensor may be used to call up a soldier's medical history,giving information immediately on allergies, blood type, and other timesensitive and treatment determining data quickly and easily, thusallowing proper treatment to be provided under battlefield conditions.This is especially helpful for patients who may be unconscious wheninitially treated and who may be missing identification tags.

A further embodiment of a device for capturing biometric data fromindividuals may incorporate a server to store and process biometric datacollected. The biometric data captured may include a hand image withmultiple fingers, a palm print, a face camera image, an iris image, anaudio sample of an individual's voice, and a video of the individual'sgait or movement. The collected data must be accessible to be useful.

Processing of the biometric data may be done locally or remotely at aseparate server. Local processing may offer the option to capture rawimages and audio and make the information available on demand from acomputer host over a WiFi or USB link. As an alternative, another localprocessing method processes the images and then transmits the processeddata over the Internet. This local processing includes the steps offinding the finger prints, rating the finger prints, finding the faceand then cropping it, finding and then rating the iris, and othersimilar steps for audio and video data. While processing the datalocally requires more complex code, it does offer the advantage ofreduced data transmission over the Internet.

A scanner associated with the biometric data collection devices may usecode that is compliant with the USB Image Device protocol that is acommonly used scanner standard. Other embodiments may use differentscanner standards, depending on need.

When a WiFi network is used to transfer the data, the Bio-Print device,which is further described herein, can function or appear like a webserver to the network. Each of the various types of images may beavailable by selecting or clicking on a web page link or button from abrowser client. This web server functionality may be part of theBio-Print device, specifically, included in the microcomputerfunctionality.

A web server may be a part of the Bio-Print microcomputer host, allowingfor the Bio-Print device to author a web page that exposes captured dataand also provides some controls. An additional embodiment of the browserapplication could provide controls to capture high resolution handprints, face images, iris images, set the camera resolution, set thecapture time for audio samples, and also enable a streaming connection,using a web cam, Skype, or similar mechanism. This connection could beattached to the audio and face camera.

A further embodiment provides a browser application that gives access toimages and audio captured via file transfer protocol (FTP) or otherprotocol. A still further embodiment of the browser application mayprovide for automatic refreshes at a selectable rate to repeatedly grabpreview images.

An additional embodiment provides local processing of captured biometricdata using a microcomputer and provides additional controls to display arating of the captured image, allowing a user to rate each of the printsfound, retrieve faces captured, and also to retrieve cropped iris imagesand allow a user to rate each of the iris prints.

Yet another embodiment provides a USB port compatible with the OpenMultimedia Application Platform (OMAP3) system. OMAP3 is a proprietarysystem on a chip for portable multimedia applications. The OMAP3 deviceport is equipped with a Remote Network Driver Interface Specification(RNDIS), a proprietary protocol that may be used on top of USB. Thesesystems provide the capability that when a Bio-Print device is pluggedinto a Windows PC USB host port, the device shows up as an IP interface.This IP interface would be the same as over WiFi (TCP/IP web server).This allows for moving data off the microcomputer host and provides fordisplay of the captured print.

An application on the microcomputer may implement the above by receivingdata from an FPGA over the USB bus. Once received, JPEG content iscreated. This content may be written over a socket to a server runningon a laptop, or be written to a file. Alternately, the server couldreceive the socket stream, pop the image, and leave it open in a window,thus creating a new window for each biometric capture. If themicrocomputer runs Network File System (NFS), a protocol for use withSun-based systems or SAMBA, a free software reimplementation thatprovides file and print services for Windows clients, the files capturedmay be shared and accessed by any client running NFS or SystemManagement Bus (SMB), a PC communication bus implementation. In thisembodiment, a JPEG viewer would display the files. The display clientcould include a laptop, augmented reality glasses, or a phone runningthe Android platform.

An additional embodiment provides for a server-side application offeringthe same services described above.

An alternative embodiment to a server-side application displays theresults on the augmented reality glasses.

A further embodiment provides the microcomputer on a removable platform,similar to a mass storage device or streaming camera. The removableplatform also incorporates an active USB serial port.

In embodiments, the eyepiece may include audio and/or visual sensors tocapture sounds and/or visuals from 360 degrees around the wearer of aneyepiece. This may be from sensors mounted on the eyepiece itself, orcoupled to sensors mounted on a vehicle that the wearer is in. Forinstance, sound sensors and/or cameras may be mounted to the outside ofa vehicle, where the sensors are communicatively coupled to the eyepieceto provide a surround sound and/or sight ‘view’ of the surroundingenvironment. In addition, the sound system of the eyepiece may providesound protection, canceling, augmentation, and the like, to help improvethe hearing quality of the wearer while they are surrounded byextraneous or loud noise. In an example, a wearer may be coupled tocameras mounted on the vehicle they are driving. These cameras may thenbe in communication with the eyepiece, and provide a 360-degree viewaround the vehicle, such as provided in a projected graphical imagethrough the eyepiece display to the wearer.

In an example, and referring to FIG. 69, control aspects of the eyepiecemay include a remote device in the form of a watch controller 6902, suchas including a receiver and/or transmitter for interfacing with theeyepiece for messaging and/or controlling the eyepiece when the user isnot wearing the eyepiece. The watch controller may include a camera, afingerprint scanner, discrete control buttons, 2D control pad, an LCDscreen, a capacitive touch screen for multi-touch control, a shakemotor/piezo bumper to give tactile feedback, buttons with tactile feel,Bluetooth, camera, fingerprint scanner, accelerometer, and the like,such as provided in a control function area 6904 or on other functionalportions 6910 of the watch controller 6902. For instance, a watchcontroller may have a standard watch display 6908, but additionally havefunctionality to control the eyepiece, such as through control functions6914 in the control function area 6904. The watch controller may displayand/or otherwise notify the user (e.g. vibration, audible sounds) ofmessages from the eyepiece, such an email, advertisements, calendaralerts, and the like, and show the content of the message that comes infrom the eyepiece that the user is currently not wearing. A shake motor,piezo bumper, and the like, may provide tactile feedback to the touchscreen control interface. The watch receiver may be able to providevirtual buttons and clicks in the control function area 6904 userinterface, buzz and bump the user's wrist, and the like, when a messageis received. Communications connectivity between the eyepiece and thewatch receiver may be provided through Bluetooth, WiFi, Cell network, orany other communications interface known to the art. The watchcontroller may utilize an embedded camera for videoconferencing (such asdescribed herein), iris scanning (e.g. for recording an image of theiris for storage in a database, for use in authentication in conjunctionwith an existing iris image in storage, and the like), picture taking,video, and the like. The watch controller may have a fingerprintscanner, such as described herein. The watch controller, or any othertactile interface described herein, may measure a user's pulse, such asthrough a pulse sensor 6912 (which may be located in the band, on theunderside of the main body of the watch, and the like. In embodiments,the eyepiece and other control/tactile interface components may havepulse detection such that the pulse from different control interfacecomponents are monitored in a synchronized way, such as for health,activity monitoring, authorization, and the like. For example, a watchcontroller and the eyepiece may both have pulse monitoring, where theeyepiece is capable of sensing whether the two are in synchronization,if both match a previously measured profile (such as forauthentication), and the like. Similarly, other biometrics may be usedfor authentication between multiple control interfaces and the eyepiece,such as with fingerprints, iris scans, pulse, health profile, and thelike, where the eyepiece knows whether the same person is wearing theinterface component (e.g. the watch controller) and the eyepiece.Biometric/health of a person may be determined by looking at IR LED viewof the skin, for looking at subsurface pulse, and the like. Inembodiments, multi-device authentication (e.g. token for Bluetoothhandshake) may be used, such as using the sensors on both devices (e.g.fingerprint on both devices as a hash for the Bluetooth token), and thelike.

Referring to FIGS. 70A-70D, the eyepiece may be stored in an eyepiececarrying case, such as including a recharge capability, an integrateddisplay, and the like. FIG. 70A depicts an embodiment of a case, shownclosed, with integrated recharge AC plug and digital display, and FIG.70B shows the same embodiment case open. FIG. 70C shows anotherembodiment case closed, and FIG. 70D shows the same embodiment open,where a digital display is shown through the cover. In embodiments, thecase may have the ability to recharge the eyepiece while in the case,such as through an AC connection or battery (e.g. a rechargeablelithium-ion battery built into the carrying case for charging theeyepiece while away from AC power). Electrical power may be transferredto the eyepiece through a wired or wireless connection, such as though awireless induction pad configuration between the case and the eyepiece.In embodiments, the case may include a digital display in communicationswith the eyepiece, such as through Bluetooth wireless, and the like. Thedisplay may provide information about the state of the eyepiece, such asmessages received, battery level indication, notifications, and thelike.

Referring to FIG. 71, the eyepiece 7120 may be used in conjunction withan unattended ground sensor unit 7102, such as formed as a stake 7104that can be inserted in the ground 7118 by personnel, fired from aremote control helicopter, dropped by plane, and the like. The groundsensor unit 7102 may include a camera 7108, a controller 7110, a sensor7112, and the like. Sensors 7112 may include a magnetic sensor, soundsensor, vibration sensor, thermal sensor, passive IR sensor, motiondetector, GPS, real-time clock, and the like, and provide monitoring atthe location of the ground sensor unit 7102. The camera 7108 may have afield of view 7114 in both azimuth and elevation, such as a full orpartial 360-degree camera array in azimuth and +/−90 degrees inelevation. The ground sensor unit 7102 may capture a sensor and imagedata of an event(s) and transmit it over a wireless network connectionto an eyepiece 7120. Further, the eyepiece may then transmit the data toan external communications facility 7122, such as a cell network, asatellite network, a WiFi network, to another eyepiece, and the like. Inembodiments, ground sensor units 7102 may relay data from unit to unit,such as from 7102A to 7102B to 7102C. Further, the data may then berelayed from eyepiece 7120A to eyepiece 7120B and on to thecommunications facility 7122, such as in a backhaul data network. Datacollected from a ground sensor unit 7102, or array of ground sensorunits, may be shared with a plurality of eyepieces, such as fromeyepiece to eyepiece, from the communications facility to the eyepiece,and the like, such that users of the eyepiece may utilize and share thedata, either in it's raw form or in a post-processed form (i.e. as agraphic display of the data through the eyepiece). In embodiments, theground sensor units may be inexpensive, disposable, toy-grade, and thelike. In embodiments, the ground sensor unit 7102 may provide backup forcomputer files from the eyepiece 7120.

Referring to FIG. 72, the eyepiece may provide control throughfacilities internal and external to the eyepiece, such as initiated fromthe surrounding environment 7202, input devices 7204, sensing devices7208, user action capture devices 7210, internal processing facilities7212, internal multimedia processing facilities, internal applications7214, camera 7218, sensors 7220, earpiece 7222, projector 7224, througha transceiver 7228, through a tactile interface 7230, from externalcomputing facilities 7232, external applications 7234, event and/or datafeeds 7238, external devices 7240, third parties 7242, and the like.Command and control modes 7260 of the eyepiece may be initiated bysensing inputs through input devices 7244, user action 7248, externaldevice interaction 7250, reception of events and/or data feeds 7252,internal application execution 7254, external application execution7258, and the like. In embodiments, there may be a series of stepsincluded in the execution control, including at least combinations oftwo of the following: events and/or data feeds, sensing inputs and/orsensing devices, user action capture inputs and/or outputs, usermovements and/or actions for controlling and/or initiating commands,command and/or control modes and interfaces in which the inputs may bereflected, applications on the platform that may use commands to respondto inputs, communications and/or connection from the on-platforminterface to external systems and/or devices, external devices, externalapplications, feedback 7262 to the user (such as related to externaldevices, external applications), and the like.

In embodiments, events and/or data feeds may include email, militaryrelated communications, calendar alerts, security events, safety events,financial events, personal events, a request for input, instruction,entering an activity state, entering a military engagement activitystate, entering a type of environment, entering a hostile environment,entering a location, and the like, and combinations of the same.

In embodiments, sensing inputs and/or sensing devices may include acharge-coupled device, black silicon sensor, IR sensor, acoustic sensor,induction sensor, motion sensor, optical sensor, opacity sensor,proximity sensor, inductive sensor, Eddy-current sensor, passiveinfrared proximity sensor, radar, capacitance sensor, capacitivedisplacement sensor, hall-effect sensor, magnetic sensor, GPS sensor,thermal imaging sensor, thermocouple, thermistor, photoelectric sensor,ultrasonic sensor, infrared laser sensor, inertial motion sensor, MEMSinternal motion sensor, ultrasonic 3D motion sensor, accelerometer,inclinometer, force sensor, piezoelectric sensor, rotary encoders,linear encoders, chemical sensor, ozone sensor, smoke sensor, heatsensor, magnetometer, carbon dioxide detector, carbon monoxide detector,oxygen sensor, glucose sensor, smoke detector, metal detector, rainsensor, altimeter, GPS, detection of being outside, detection ofcontext, detection of activity, object detector (e.g. billboard), markerdetector (e.g. geo-location marker for advertising), laser rangefinder,sonar, capacitance, optical response, heart rate sensor, RF/micropowerimpulse radio (MIR) sensor, and the like, and combinations of the same.

In embodiments, user action capture inputs and/or devices may include ahead tracking system, camera, voice recognition system, body movementsensor (e.g. kinetic sensor), eye-gaze detection system, tongue touchpad, sip-and-puff systems, joystick, cursor, mouse, touch screen, touchsensor, finger tracking devices, 3D/2D mouse, inertial movementtracking, microphone, wearable sensor sets, robotic motion detectionsystem, optical motion tracking system, laser motion tracking system,keyboard, virtual keyboard, virtual keyboard on a physical platform,context determination system, activity determination system (e.g. on atrain, on a plane, walking, exercising, etc.) finger following camera,virtualized in-hand display, sign language system, trackball,hand-mounted camera, temple-located sensors, glasses-located sensors,Bluetooth communications, wireless communications, satellitecommunications, and the like, and combinations of the same.

In embodiments, user movements or actions for controlling or initiatingcommands may include head movement, head shake, head nod, head roll,forehead twitch, ear movement, eye movement, eye open, eye close, blinkon eye, eye roll, hand movement, clench fist, open fist, shake fist,advance fist, retract fist, voice commands, sip or puff on straw, tonguemovement, finger movement, one or more finger movements, extend fingercrook finger, retract finger, extend thumb, make symbol with finger(s),make symbol with finger and thumb, depress finger of thumb, drag anddrop with fingers, touch and drag, touch and drag with two fingers,wrist movement, wrist roll, wrist flap, arm movement, arm extend, armretract, arm left turn signal, arm right turn signal, arms akimbo, armsextended, leg movement, leg kick, leg extend, leg curl, jumping jack,body movement walk, run turn left, turn right, about-face, twirl, armsup and twirl, arms down and twirl, one left out and twirl, twirl withvarious hand and arm positions, finger pinch and spread motions, fingermovement (e.g. virtual typing), snapping, tapping hip motion, shouldermotion foot motions, swipe movements, sign language (e.g. ASL), and thelike, and combinations of the same.

In embodiments, command and/or control modes and interfaces in whichinputs can be reflected may include a graphical user interface (GUI),auditory command interface, clickable icons, navigable lists, virtualreality interface, augmented reality interface, heads-up display,semi-opaque display, 3D navigation interface, command line, virtualtouch screen, robot control interface, typing (e.g. with persistentvirtual keyboard locked in place), predictive and/or learning based userinterface (e.g. learns what the wearer does in a ‘training mode’, andwhen and where they do it), simplified command mode (e.g. hand gesturesto kick off an application, etc), Bluetooth controllers, cursor hold,lock a virtual display, head movement around a located cursor, and thelike, and combinations of the same.

In embodiments, applications on the eyepiece that can use commandsand/or respond to inputs may include military applications, weaponscontrol applications, military targeting applications, war gamesimulation, hand-to-hand fighting simulator, repair manual applications,tactical operations applications, mobile phone applications (e.g. iPhoneapps), information processing, fingerprint capture, facial recognition,information display, information conveying, information gathering, iriscapture, entertainment, easy access to information for pilots, locatingobjects in 3D in the real world, targeting for civilians, targeting forpolice, instructional, tutorial guidance without using hands (e.g. inmaintenance, assembly, first aid, etc), blind navigation assistance,communications, music, search, advertising, video, computer games,video, computer games, eBooks, advertising, shopping, e-commerce,videoconferencing, and the like, and combinations of the same.

In embodiments, communication and/or connection from the eyepieceinterface to external systems and devices may include a microcontroller,microprocessor, digital signal processor, steering wheel controlinterface, joystick controller, motion and sensor resolvers, steppercontroller, audio system controller, program to integrate sound andimage signals, application programming interface (API), graphical userinterface (GUI), navigation system controller, network router, networkcontroller, reconciliation system, payment system, gaming device,pressure sensor, and the like.

In embodiments, external devices to be controlled may include a weapon,a weapon control system, a communications system, a bomb detectionsystem, a bomb disarming system, a remote-controlled vehicle, a computer(and thus many devices able to be controlled by a computer), camera,projector, cell phone, tracking devices, display (e.g. computer, video,TV screen), video game, war game simulator, mobile gaming, pointing ortracking device, radio or sound system, range finder, audio system,iPod, smart phone, TV, entertainment system, computer controlled weaponssystem, drone, robot, automotive dashboard interfaces, lighting devices(e.g. mood lighting), exercise equipment, gaming platform (such as thegaming platform recognizing the user and preloading what they like toplay), vehicles, storage-enabled devices, payment system, ATM, POSsystem, and the like.

In embodiments, applications in association with external devices may bemilitary applications, weapons control applications, military targetingapplications, war game simulation, hand-to-hand fighting simulator,repair manual applications, tactical operations applications,communications, information processing, fingerprint capture, facialrecognition, iris capture, entertainment, easy access to information forpilots, locating objects in 3D in the real world, targeting forcivilians, targeting for police, instructional, tutorial guidancewithout using hands (e.g. maintenance, assembly, first aid), blindnavigation assistance, music, search, advertising, video, computergames, eBooks, automotive dashboard applications, advertising, militaryenemy targeting, shopping, e-commerce, and the like, and combinations ofsame.

In embodiments, feedback to the wearer related to external devices andapplications may include visual display, heads-up display, bulls-eye ortarget tracking display, tonal output or sound warning, performance orrating indicator, score, mission accomplished indication, actioncomplete indication, play of content, display of information, reports,data mining, recommendations, targeted advertisements, and the like.

In an example, control aspects of the eyepiece may include combinationsof a head nod from a soldier as movement to initiate a silent command(such as during a combat engagement), through a graphical user interfacefor reflecting modes and/or interfaces in which the control input isreflected, a military application on the eyepiece that uses the commandsand/or responds to the control input, an audio system controller tocommunicate and/or connect from the eyepiece interface to an externalsystem or device, and the like. For instance, the soldier may becontrolling a secure communications device through the eyepiece during acombat engagement, and wish to change some aspect of communications,such as a channel, a frequency, an encoding level, and the like, withoutmaking a sound and with minimal motion so as to minimize the chance ofbeing heard or seen. In this instance, a nod of the soldier's head maybe programmed to indicate the change, such as a quick nod forward toindicate the beginning of a transmission, a quick nod backward toindicate the end of a transmission, and the like. In addition, theeyepiece may be projecting a graphical user interface to the soldier forthe secure communications device, such as showing what channel isactive, what alternative channels are available, others in their teamthat are currently transmitting, and the like. The nod of the soldiermay then be interpreted by processing facilities of the eyepiece as achange command, the command transmitted to the audio system controller,and the graphical user interface for the communications device showingthe change. Further, certain nods/body motions may be interpreted asspecific commands to be transmitted such that the eyepiece sends apre-established communication without the soldier needing to be audible.That is, the soldier may be able to send pre-canned communications totheir team though body motions (for example, as determined together withthe team prior to the engagement). In this way, a soldier wearing andutilizing the facilities of the eyepiece may be able to connect andinterface with the external secure communications device in a completelystealthy manner, maintaining silent communications with their teamduring engagement, even when out of sight with the team. In embodiments,other movements or actions for controlling or initiating commands,command and/or control modes and interfaces in which the inputs can bereflected, applications on platform that can use commands and/or respondto inputs, communication or connection from the on-platform interface toexternal systems and devices, and the like, as described herein, mayalso be applied.

In an example, control aspects of the eyepiece may include combinationsof motion and position sensors as sensing inputs, an augmented realityinterface as a command and control interface in which the inputs can bereflected to a soldier, a motion sensor and range finder for a weaponsystem as external devices to be controlled and information collectedfrom, feedback to the soldier related to the external devices, and thelike. For instance, a soldier wearing the eyepiece may be monitoringmilitary movements within an environment with the motion sensor, andwhen the motion sensor is triggered an augmented reality interface maybe projected to the wearer that helps identify a target, such as aperson, vehicle, and the like for further monitoring and/or targeting.In addition, the range finder may be able to determine the range to theobject and feedback that information to the soldier for use in targeting(such as manually, with the soldier executing a firing action; orautomatically, with the weapon system receiving the information fortargeting and the soldier providing a command to fire). In embodiments,the augmented reality interface may provide information to the soldierabout the target, such as the location of the object on a 2D or 3Dprojected map, identity of the target from previously collectedinformation (e.g. as stored in an object database, including facerecognition, object recognition), coordinates of the target, nightvision imaging of the target, and the like. In embodiments, thetriggering of the motion detector may be interpreted by processingfacilities of the eyepiece as a warning event, the command may betransmitted to the range finder to determine the location of the object,as well as to the speakers of the ear phones of the eyepiece to providean audio warning to the soldier that a moving object has been sensed inthe area being monitored. The audio warning plus visual indicators tothe soldier may serve as inputs to the soldier that attention should bepaid to the moving object, such as if the object has been identified asan object of interest to the soldier, such as through an accesseddatabase for known combatants, known vehicle types, and the like. Forinstance, the soldier may be at a guard post monitoring the perimeteraround the post at night. In this case, the environment may be dark, andthe soldier may have fallen into a low attentive state, as it may belate at night, with all environmental conditions quiet. The eyepiece maythen act as a sentry augmentation device, ‘watching’ from the soldier'spersonal perspective (as opposed to some external monitoring facilityfor the guard post). When the eyepiece senses movement, the soldier maybe instantly alerted as well as guided to the location, range, identity,and the like, of the motion. In this way, the soldier may be able toreact to avoid personal danger, to target fire to the located movement,and the like, as well as alert the post to potential danger. Further, ifa firelight were to ensue, the soldier may have improved reaction timeas a result of the warning from the eyepiece, with better decisionmaking though information about the target, and minimizing the danger ofbeing injured or the guard post from being infiltrated. In embodiments,other sensing inputs and/or sensing devices, command and/or controlmodes and interfaces in which the inputs can be reflected, usefulexternal devices to be controlled, feedback related to external devicesand/or external applications, and the like, as described herein, mayalso be applied.

In embodiments, the eyepiece may enable remote control of vehicles, suchas a truck, robot, drone, helicopter, watercraft, and the like. Forinstance, a soldier wearing the eyepiece may be able to command throughan internal communications interface for control of the vehicle. Vehiclecontrol may be provided through voice commands, body movement (e.g. asoldier instrumented with movement sensors that are in interactivecommunication with the eyepiece, and interfaced through the eyepiece tocontrol the vehicle), keyboard interface, and the like. In an example, asoldier wearing an eyepiece may provide remote control to a bombdisposal robot or vehicle, where commands are generated by the soldierthough a command interface of the eyepiece, such as described herein. Inanother example, a soldier may command an aircraft, such as a remotecontrol drone, remote control tactical counter-rotating helicopter, andthe like. Again, the soldier may provide control of the remote controlaircraft through control interfaces as described herein.

In an example, control aspects of the eyepiece may include combinationsof a wearable sensor set as an action capture input for a soldier,utilizing a robot control interface as a command and control interfacein which the inputs can be reflected, a drone or other robotic device asan external device to be controlled, and the like. For instance, thesoldier wearing the eyepiece may be instrumented with a sensor set forthe control of a military drone, such as with motion sensor inputs tocontrol motion of the drone, hand recognition control for manipulationof control features of the drone (e.g. such as through a graphical userinterface displayed through the eyepiece), voice command inputs forcontrol of the drone, and the like. In embodiments, control of the dronethrough the eyepiece may include control of flight, control of on-boardinterrogation sensors (e.g. visible camera, IR camera, radar), threatavoidance, and the like. The soldier may be able to guide the drone toits intended target using body mounted sensors and picturing the actualbattlefield through a virtual 2D/3D projected image, where flight,camera, monitoring controls are commanded though body motions of thesoldier. In this way, the soldier may be able to maintain anindividualistic, full visual immersion, of the flight and environment ofthe drone for greater intuitive control. The eyepiece may have a robotcontrol interface for managing and reconciling the various controlinputs from the soldier-worn sensor set, and for providing an interfacefor control of the drone. The drone may then be controlled remotelythrough physical action of the soldier, such as through a wirelessconnection to a military control center for drone control andmanagement. In another similar example, a soldier may control abomb-disarming robot that may be controlled through a soldier-wornsensor set and associated eyepiece robot control interface. Forinstance, the soldier may be provided with a graphical user interfacethat provides a 2D or 3D view of the environment around the bombdisarming robot, and where the sensor pack provides translation of themotion of the soldier (e.g. arms, hands, and the like) to motions of therobot. In this way, the soldier may be able to provide a remote controlinterface to the robot to better enable sensitive control during thedelicate bomb disarming process. In embodiments, other user actioncapture inputs and/or devices, command and/or control modes andinterfaces in which the inputs can be reflected, useful external devicesto be controlled, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof an event indication to the soldier as they enter a location, apredictive-learning based user interface as a command and control modeand/or interface in which the input occurrence of the event isreflected, a weapons control system as an external device to becontrolled, and the like. For instance, an eyepiece may be programmed tolearn the behavior of a soldier, such as what the soldier typically doeswhen they enter a particular environment with a particular weaponscontrol system, e.g. does the wearer turn on the system, arm the system,bring up visual displays for the system, and the like. From this learnedbehavior, the eyepiece may be able to make a prediction of what thesoldier wants in the way of an eyepiece control function. For example,the soldier may be thrust into a combat situation, and needs theimmediate use of a weapons control system. In this case, the eyepiecemay sense the location and/or the identity of the weapons system as thesoldier approaches, and configure/enable the weapons system to how thesoldier typically configures the system when they are near the weaponscontrol system, such as in previous uses of the weapons system where theeyepiece was in a learning mode, and commanding the weapons controlsystem to turn on the system as last configured. In embodiments, theeyepiece may sense the location and/or identity of the weapons systemthrough a plurality of methods and systems, such as through a visionsystem recognizing the location, an RFID system, a GPS system, and thelike. In embodiments, the commanding of the weapons control system maybe through a graphical user interface that provides the soldier with avisual for fire-control of the weapon system, an audio-voice commandsystem interface that provides choices to the soldier and voicerecognition for commanding, pre-determined automatic activation of afunction, and the like. In embodiments, there may be a profileassociated with such learned commanding, where the soldier is able tomodify the learned profile and/or set preferences within the learnedprofile to help optimize automated actions, and the like. For example,the soldier may have separate weapon control profiles for weaponsreadiness (i.e. while on post and awaiting action) and for activeweapons engagement with the enemy. The soldier may need to modify aprofile to adjust to changing conditions associated with use of theweapon system, such as a change in fire command protocols, ammunitiontype, added capabilities of the weapon system, and the like. Inembodiments, other events and/or data feeds, command and/or controlmodes and interfaces in which the inputs can be reflected, usefulexternal devices to be controlled, and the like, as described herein,may also be applied.

In an example, control aspects of the eyepiece may include combinationsof an individual responsibility event for a soldier (such as deployed ina theater of action, and managing their time) as an event and/or datafeed, a voice recognition system as a user action capture input device,an auditory command interface as a command and control interface inwhich the inputs can be reflected, video-based communications as anapplication on the eyepiece that is used to respond to the input fromthe soldier, and the like. For instance, a soldier wearing the eyepiecemay get a visual indication projected to them of a scheduled event for agroup video supported communication between commanders. The soldier maythen use a voice command to an auditory command interface on theeyepiece to bring up the contact information for the call, and voicecommand the group video communication to be initiated. In this way, theeyepiece may serve as a personal assistant for the soldier, bringing upscheduled events and providing the soldier with a hands-free commandinterface to execute the scheduled events. In addition, the eyepiece mayprovide for the visual interface for the group video communication,where the images of the other commanders are projected to the soldierthrough the eyepiece, and where an external camera is providing thesoldier's video image through communicative connection to the eyepiece(such as with an external device with a camera, using a mirror with theinternally integrated camera, and the like, as described herein). Inthis way, the eyepiece may provide a fully integrated personal assistantand phone/video-based communications platform, subsuming the functionsof other traditionally separate electronics devices, such as the radio,mobile phone, a video-phone, a personal computer, a calendar, ahands-free command and control interface, and the like. In embodiments,other events and/or data feeds, user action capture inputs and/ordevices, command and/or control modes and interfaces in which the inputscan be reflected, applications on platform that can use commands and/orrespond to inputs, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof a security event to a soldier as an event and/or data feed; a cameraand touch screen as user action capture input devices; an informationprocessing, fingerprint capture, facial recognition application on theeyepiece to respond to the inputs; a graphical user interface forcommunications and/or connection between the eyepiece and externalsystems and devices; and an external information processing, fingerprintcapture, facial recognition application and database for access toexternal security facilities and connectivity, and the like. Forinstance, a soldier may receive a ‘security event’ while on post at amilitary checkpoint where a plurality of individuals is to be securitychecked and/or identified. In this case there may be a need forrecording the biometrics of the individuals, such as because they don'tshow up in a security database, because of suspicious behavior, becausethey fit the profile of a member of a combatant, and the like. Thesoldier may then use biometric input devices, such as a camera forphotographing faces and a touch screen for recording fingerprints, wherethe biometric inputs are managed though an internal information,processing, fingerprint capture, and facial recognition application onthe eyepiece. In addition, the eyepiece may provide a graphical userinterface as a communications connection to an external information,processing, fingerprint capture, and facial recognition application,where the graphical user interface provides data capture interfaces,external database access, people of interest database, and the like. Theeyepiece may provide for an end-to-end security management facility,including monitoring for people of interest, input devices for takingbiometric data, displaying inputs and database information, connectivityto external security and database applications, and the like. Forinstance, the soldier may be checking people through a militarycheckpoint, and the soldier has been commanded to collect facial images,such as with iris biometrics, for anyone that meets a profile and is notcurrently in a security database. As individuals approach the soldier,as in a line to pass through the checkpoint, the soldier's eyepiecetakes high-resolution images of each individual for facial and/or irisrecognition, such as checked though a database accessible though anetwork communication link A person may be allowed to pass thecheckpoint if they do not meet the profile (e.g. a young child), or isin the database with an indication that they are not considered athreat. A person may not be allowed to pass through the checkpoint, andis pulled aside, if the individual is indicated to be a threat or meetsthe profile and is not in the database. If they need to be entered intothe security database, the soldier may be able to process the individualdirectly through facilities of the eyepiece or with the eyepiececontrolling an external device, such as for collecting personalinformation for the individual, taking a close-up image of theindividual's face and/or iris, recording fingerprints, and the like,such as described herein. In embodiments, other events and/or datafeeds, user action capture inputs and/or devices, applications onplatform that can use commands and/or respond to inputs, communicationor connection from the on-platform interface to external systems anddevices, applications for external devices, and the like, as describedherein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof a finger movement as a user action for a soldier initiating aneyepiece command, a clickable icon as a command and control mode and/orinterface in which the user action can be reflected, an application onthe eyepiece (e.g. weapons control, troop movements, intelligence datafeed, and the like), a military application tracking API as acommunication and/or connection from the eyepiece application to anexternal system, an external personnel tracking application, feedback tomilitary personnel, and the like. For instance, a system for monitoringa soldier's selection of an on-eyepiece application may be implementedthrough an API such that the monitoring provides a service to themilitary for monitoring and tracking application usage, feedback to thesoldier as to other applications available to them based on themonitored behavior, and the like. In the course of a day, the soldiermay select an application for use and/or download, such as through agraphical user interface where clickable icons are presented, and towhich the soldier may be able to select the icon based on a fingermovement control implementation facility (such as a camera or inertialsystem through which the soldier's finger action is used as a controlinput, in this case to select the clickable icon). The selection maythen be monitored through the military application tracking API thatsends the selection, or stored number of selections (such astransmitting stored selections over a period of time), to the externalpersonnel tracking application. The soldier's application selections, inthis case ‘virtual clicks’, may then be analyzed for the purpose ofoptimizing usage, such as through increasing bandwidth, change ofavailable applications, improvement to existing applications, and thelike. Further, the external personnel tracking application may utilizethe analysis to determine what the wearer's preferences are in terms ofapplications use, and send the wearer feedback in the form ofrecommendations of applications the wearer may be interested in, apreference profile, a list of what other similar military users areutilizing, and the like. In embodiments, the eyepiece may provideservices to improve the soldier's experience with the eyepiece, such aswith recommendations for usage that the soldier may benefit from, andthe like, while aiding in guiding the military use of the eyepiece andapplications thereof. For instance, a soldier that is new to using theeyepiece may not fully utilize its capabilities, such as in use ofaugmented reality interfaces, organizational applications, missionsupport, and the like. The eyepiece may have the capability to monitorthe soldier's utilization, compare the utilization to utilizationmetrics (such as stored in an external eyepiece utilization facility),and provide feedback to the soldier in order to improve use andassociated efficiency of the eyepiece, and the like. In embodiments,other user movements or actions for controlling or initiating commands,command and/or control modes and interfaces in which the inputs can bereflected, applications on platform that can use commands and/or respondto inputs, communication or connection from the on-platform interface toexternal systems and devices, applications for external devices,feedback related to external devices and/or external applications, andthe like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof body movement (e.g. kinetic sensor) and touch sensors as user actioncapture sensing devices, head and hand movement as user actions forcontrolling and/or initiating commands, a virtual reality interface as acommand and control interface through which the inputs can be reflected,an information display as an application on the eyepiece that canrespond to the inputs, a combat simulator as an external device to becontrolled through a combat simulation application, and the activationof the combat simulator content to the soldier with performance, rating,score, and the like, as feedback to the user related to the externaldevice and application. For instance, a soldier may be able to interactwith an artificial reality enhanced combat simulator, where the wearer'sbody movements are interpreted as control inputs, such as though bodymovement sensors, touch sensors, and the like. In this way, movements ofthe wearer's body may be fed into the combat simulator, rather thanusing more traditional control inputs such as a handheld controller.Thus, the soldier's experience may be more realistic, such as to providebetter muscle memory from the simulated combat exercise, such as whenengaged in defensive avoidance, in a firelight, and the like, and wherethe eyepiece provides a full immersion experience for the soldierwithout the need for external devices that would normally not be used bythe soldier in a live action. Body motion control inputs may feed into avirtual reality interface and information display application on theeyepiece to provide the user with the visual depiction of the simulatedcombat environment. In embodiments, the combat simulator may be runentirely on-board the eyepiece as a local application, interfaced to anexternal combat simulator facility local to the wearer, interfaced to anetworked combat simulator facility (e.g. a massively multiplayer combatsimulator, an individual combat simulator, a group combat simulatorthrough a local network connection), and the like. In the case where theeyepiece is interfacing and controlling a hybrid local-external combatsimulator environment, the eyepiece application portion of simulationexecution may provide the visual environment and information display tothe soldier, and the external combat simulator facility may provide thecombat simulator application execution. It would be clear to one skilledin the art that many different partitioning configurations between theprocessing provided by the eyepiece and processing provided by externalfacilities may be implemented. Further, the combat simulatorimplementation may extend to external facilities across a securenetwork. External facilities, whether local or across the securenetwork, may then provide feedback to the soldier, such as in providingat least a portion of the executed content (e.g. the locally providedprojection combined with content from the external facilities and othersoldiers), performance indications, scores, rankings, and the like. Inembodiments, the eyepiece may provide a soldier environment where theeyepiece interfaces with external control inputs and external processingfacilities, to create the next generation of combat simulator platform.In embodiments, other sensing inputs and/or sensing devices, usermovements or actions for controlling or initiating commands, commandand/or control modes and interfaces in which the inputs can bereflected, applications on platform that can use commands and/or respondto inputs, useful external devices to be controlled, feedback related toexternal devices and/or external applications, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof IR, thermal, force, carbon monoxide, and the like sensors as inputs;microphone as an additional input device; voice commands as an action bya soldier to initiate commands; a heads-up display as a command andcontrol interface in which the inputs can be reflected; an instructionalguidance application to provide guidance while reducing the need for thesoldier to use their hands, such as in emergency repair in the field,maintenance, assembly, and the like; a visual display that providesfeedback to the soldier based on the actions of the soldier and thesensor inputs; and the like. For instance, a soldier's vehicle may havebeen damaged in a firefight, leaving the soldier(s) stranded withoutimmediate transport capabilities. The soldier may be able to bring up aninstructional guidance application, as running through the eyepiece, toprovide hands-free instruction and computer-based expert knowledgeaccess to diagnosing the problem with the vehicle. In addition, theapplication may provide a tutorial for procedures not familiar to thesoldier, such as restoring basic and temporary functionality of thevehicle. The eyepiece may also be monitoring various sensor inputsrelevant to the diagnosis, such as an IR, thermal, force, ozone, carbonmonoxide, and the like sensors, so that the sensor input may beaccessible to the instructional application and/or directly accessibleto the soldier. The application may also provide for a microphonethrough which voice commands may be accepted; a heads-up display for thedisplay of instruction information, 2D or 3D depiction of the portion ofthe vehicle under repair; and the like. In embodiments, the eyepiece maybe able to provide a hands-free virtual assistant to the soldier toassist them in the diagnosis and repair of the vehicle in order tore-establish a means for transport, allowing the soldier to re-engagethe enemy or move to safety. In embodiments, other sensing inputs and/orsensing devices, user action capture inputs and/or devices, usermovements or actions for controlling or initiating commands, commandand/or control modes and interfaces in which the inputs can bereflected, applications on platform that can use commands and/or respondto inputs, feedback related to external devices and/or externalapplications, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof the eyepiece entering an ‘activity state’, such as a ‘militaryengagement’ activity mode, e.g. the soldier commanding the eyepiece intoa military engagement mode, or the eyepiece sensing it is in proximityto a military activity, perhaps even a predetermined or targetedengagement area through a received mission directive, which may havefurther been developed in part through self monitoring and learning thewearer's general engagement assignment. Continuing with this example,entering an activity state e.g. a military engagement activity state,such as while driving in a vehicle into an encounter with the enemy orinto hostile territory, may be combined with an object detector as asensing input or sensing device, a head-mounted camera and/or eye-gazedetection system as a user action capture input, eye movement as a usermovement or action for controlling or initiating commands, a 3Dnavigation interface as a command and control mode and/or interface inwhich the inputs can be reflected, an engagement management applicationon-board the eyepiece as an application for coordinating command inputsand user interface, a navigation system controller to communicate orconnect with external systems or devices, a vehicle navigation system asan external device to be controlled and/or interfaced with, a militaryplanning and execution facility as an external application forprocessing user actions with regard to a military directive, bulls-eyeor target tracking system as feedback to the wearer as to enemytargeting opportunities within sight while driving, and the like. Forinstance, a soldier may enter a hostile environment while driving theirvehicle, and the eyepiece, detecting the presence of the enemyengagement area (e.g. through GPS, direct viewing targets through anintegrated camera, and the like) may enter a ‘military engagementactivity state’ (such as enabled and/or approved by the soldier). Theeyepiece may then detect an enemy vehicle, hostile dwelling, and thelike with an object detector that locates an enemy targetingopportunity, such as through a head-mounted camera. Further, an eye-gazedetection system on the eyepiece may monitor where the soldier islooking, and possibly highlight information about a target at thelocation of the wearer's gaze, such as enemy personnel, enemy vehicle,enemy weapons, as well as friendly forces, where friend and foe areidentified and differentiated. The soldier's eye movement may also betracked, such as for changing targets of interest, or for command inputs(e.g. a quick nod indicating a selection command, a downward eyemovement indicating a command for additional information, and the like).The eyepiece may invoke a 3D navigation interface projection to assistin providing the soldier with information associated with theirsurroundings, and a military engagement application for coordinating themilitary engagement activity state, such as taking inputs from thesoldier, providing outputs to the 3D navigation interface, interfacingwith external devices and applications, and the like. The eyepiece mayfor instance utilize a navigation system controller to interface with avehicle navigation system, and thus may include the vehicle navigationsystem into the military engagement experience. Alternately, theeyepiece may use its own navigation system, such as in place of thevehicle system or to augment it, such as when the soldier gets out ofthe vehicle and wishes to have over-the-ground directions provided tothem. As part of the military engagement activity state, the eyepiecemay interface with an external military planning and execution facility,such as to provide current status, troop movements, weather conditions,friendly forces position and strength, and the like. In embodiments, thesoldier, through entering an activity state, may be provided feedbackassociated with the activity state, such as for a military engagementactivity state being supplied feedback in the form of informationassociated with an identified target. In embodiments, other eventsand/or data feeds, sensing inputs and/or sensing devices, user actioncapture inputs and/or devices, user movements or actions for controllingor initiating commands, command and/or control modes and interfaces inwhich the inputs can be reflected, applications on platform that can usecommands and/or respond to inputs, communication or connection from theon-platform interface to external systems and devices, applications forexternal devices, feedback related to external devices and/or externalapplications, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof a secure communications reception as a triggering event to a soldier,inertial movement tracking as a user action capture input device,drag-and-drop with fingers and swipe movements by the soldier as usermovements or actions for controlling or initiating commands, navigablelists as a command and control interface in which the inputs can bereflected, information conveying as a type of application on theeyepiece that can use commands and respond to inputs, a reconciliationsystem as a communication or connection from the on-eyepiece interfaceto external systems and devices, iris capture and recognition system asan external application for external systems and devices, and the like.A soldier wearing the eyepiece may receive a secure communication, andthe communication may come in to the eyepiece as an ‘event’ to thesoldier, such as to trigger an operations mode of the eyepiece, with avisual and/or audible alert, to initiate an application or action on theeyepiece, and the like. The soldier may be able to react to the eventthrough a plurality of control mechanisms, such as the wearer ‘drag anddropping’, swiping, and the like with their fingers and hands through ahand gesture interface (e.g. through a camera and hand gestureapplication on-board the eyepiece, where the wearer drags the email orinformation within the communication into a file, an application,another communication, and the like). The wearer may call up navigablelists as part of acting on the communication. The user may convey theinformation from the secure communication through an eyepieceapplication to external systems and devices, such as a reconciliationsystem for tracking communications and related actions. In embodiments,the eyepiece and/or secure access system may require identificationverification, such as through biometric identity verification e.g.fingerprint capture, iris capture recognition, and the like. Forinstance, the soldier may receive a secure communication that is asecurity alert, where the secure communication comes with secure linksto further information, and where the soldier is required to providebiometric authentication before being provided access. Onceauthenticated, the soldier may be able to use hand gestures in theirresponse and manipulation of content available through the eyepiece,such as manipulating lists, links, data, images, and the like availabledirectly from the communications and/or through the included links.Providing the capability for the soldier to respond and manipulatecontent in association with the secure communication, may better allowthe soldier to interact with the message and content in a manner thatdoes not compromise any non-secure environment they may currently be in.In embodiments, other events and/or data feeds, user action captureinputs and/or devices, user movements or actions for controlling orinitiating commands, command and/or control modes and interfaces inwhich the inputs can be reflected, applications on platform that can usecommands and/or respond to inputs, communication or connection from theon-platform interface to external systems and devices, applications forexternal devices, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using an inertial user interface as a user action capture inputdevice to provide military instruction to a soldier through the eyepieceto an external display device. For instance, a soldier, wearing theeyepiece, may wish to provide instruction to a group of other soldiersin the field from a briefing that has been made available to themthrough the facilities of the eyepiece. The soldier may be aided thoughthe use of a physical 3D or 2D mouse (e.g. with inertial motion sensor,MEMS inertial sensor, ultrasonic 3D motion sensor, IR, ultrasonic, orcapacitive tactile sensor, accelerometer, and the like), a virtualmouse, a virtual touch screen, a virtual keyboard, and the like toprovide an interface for manipulating content in the briefing. Thebriefing may be viewable to and manipulated though the eyepiece, butalso exported in real-time, such as to an external router that isconnected to an external display device (e.g. computer monitor,projector, video screen, TV screen, and the like). As such, the eyepiecemay provide a way for the soldier to have others view what they seethrough the eyepiece and as controlled through the control facilities ofthe eyepiece, allowing the soldier to export multimedia contentassociated with the briefing as enabled through the eyepiece to othernon-eyepiece wearers. In an example, a mission briefing may be providedto a commander in the field, and the commander, through the eyepiece,may be able to brief their team with multimedia and augmented realityresources available through the eyepiece, as described herein, thusgaining the benefit that such visual resources provide. In embodiments,other sensing inputs and/or sensing devices, user action capture inputsand/or devices, command and/or control modes and interfaces in which theinputs can be reflected, communication or connection from theon-platform interface to external systems and devices, useful externaldevices to be controlled, feedback related to external devices and/orexternal applications, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using an events/data feed and sensing inputs/sensing devices, such aswhere a security event plus an acoustic sensor may be implemented. Theremay be a security alert sent to a soldier and an acoustic sensor isutilized as an input device to monitor voice content in the surroundingenvironment, directionality of gunfire, and the like. For instance, asecurity alert is broadcast to all military personnel in a specificarea, and with the warning, the eyepiece activates an application thatmonitors an embedded acoustic sensor array that analyzes loud sounds toidentify the type of source for the sound, and direction from which thesound came. In embodiments, other events and/or data feeds, sensinginputs and/or sensing devices, and the like, as described herein, mayalso be applied.

In an example, control aspects of the eyepiece may include combinationsof using an events/data feed and user action capture inputs/devices,such as for a request for an input plus use of a camera. A soldier maybe in a location of interest and is sent a request for photos or videofrom their location, such as where the request is accompanied withinstructions for what to photograph. For instance, the soldier is at acheckpoint, and at some central command post it is determined that anindividual on interest may attempt to cross the checkpoint. Centralcommand may then provide instructions to eyepiece users in proximity tothe checkpoint to record and upload images and video, which may inembodiments be preformed automatically without the soldier needing tomanually turn on the camera. In embodiments, other events and/or datafeeds, user action capture inputs and/or devices, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using an events/data feed and user movements or actions forcontrolling or initiating commands, such as when a soldier is enteringan ‘activity state’ and they use a hand gesture for control. A soldiermay be put in an activity state of readiness to engage the enemy, andthe soldier uses hand gestures to silently command the eyepiece withinan engagement command and control environment. For instance, the soldiermay suddenly enter an enemy area as determined by new intelligencereceived that places the eyepiece in a heightened alert state. In thisstate it may be a requirement that silence may be required, and so theeyepiece transitions to a hand gesture command mode. In embodiments,other events and/or data feeds, user movements or actions forcontrolling or initiating commands, and the like, as described herein,may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using an events/data feed and command/control modes and interfaces inwhich the inputs can be reflected, such as entering a type ofenvironment and the user of a virtual touch screen. A soldier may entera weapons system area, and a virtual touch screen is made available tothe wearer for at least a portion of the control of the weapons system.For instance, the soldier enters a weapons vehicle, and the eyepiecedetecting the presence of the weapons system, and that the soldier isauthorized to use the weapon, brings up a virtual fire control interfacewith virtual touch screen. In embodiments, other events and/or datafeeds, command and/or control modes and interfaces in which the inputscan be reflected, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using an events/data feed and applications on platform that can usecommands/respond to inputs, such as for a safety event in combinationwith easy access to information for pilots. A military pilot (or someoneresponsible for the flight checkout of a pilotless aircraft) may receivea safety event notification as they approach an aircraft prior to theaircraft taking off, and an application is brought up to walk themthrough the pre-flight checkout. For instance, a drone specialistapproaches a drone to prepare it for launch, and an interactive checkoutprocedure is displayed to the soldier by the eyepiece. In addition, acommunications channel may be opened to the pilot of the drone so theyare included in the pre-flight checkout. In embodiments, other eventsand/or data feeds, applications on platform that can use commands and/orrespond to inputs, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using an events/data feed and a communication or connection from theon-platform interface to external systems and devices, such as thesoldier entering a location and a graphical user interface (GUI). Asoldier may enter a location where they are required to interact withexternal devices, and where the external device is interfaced throughthe GIU. For instance, a soldier gets in a military transport, and thesoldier is presented with a GUI that opens up an interactive interfacethat instructs the soldier on what they need to do during differentphases of the transport. In embodiments, other events and/or data feeds,communication or connection from the on-platform interface to externalsystems and devices, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using an events/data feed and a useful external device to becontrolled, such as for an instruction provided and a weapon system. Asoldier may be provided instructions, or a feed of instructions, whereat least one instruction pertains to the control of an external weaponssystem. For instance, a soldier may be operating a piece of artillery,and the eyepiece is providing them not only performance and proceduralinformation in association with the weapon, but also provides a feed ofinstructions, corrections, and the like, associated with targeting. Inembodiments, other events and/or data feeds, useful external devices tobe controlled, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using an events/data feed and an application for a useful externaldevice, such as in a security event/feed and biometricscapture/recognition. A soldier may be sent a security event notificationthrough (such as through a security feed) to capture biometrics(fingerprints, iris scan, walking gait profile) of certain individuals,where the biometrics are stored, evaluated, analyzed, and the like,through an external biometrics application (such as served from a securemilitary network-based server/cloud). In embodiments, other eventsand/or data feeds, applications for external devices, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using an events/data feed and feedback to a soldier related to theexternal devices and applications, such as entering an activity stateand the soldier being provided a display of information. A soldier mayplace the eyepiece into an activity state such as for military staging,readiness, action, debrief, and the like, and as feedback to beingplaced into the activity state the soldier receives a display ofinformation pertaining to the entered state. For instance, a soldierenters into a staging state for a mission, where the eyepiece fetchesinformation from a remote server as part of the tasks the soldier has tocomplete during staging, including securing equipment, additionaltraining, and the like. In embodiments, other events and/or data feeds,feedback related to external devices and/or external applications, andthe like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using sensing inputs/sensing devices and user action captureinputs/devices, such as with an inertial motion sensor and head trackingsystem. The head motion of a soldier may be tracked through inertialmotion sensor(s) in the eyepiece, such as for nod control of theeyepiece, view direction sensing for the eyepiece, and the like. Forinstance, the soldier may be a targeting a weapon system, and theeyepiece senses the gaze direction of the soldier's head through theinertial motion sensor(s) to provide continuous targeting of the weapon.Further, the weapon system may move continuously in response to thesoldier's gaze direction, and so be continuously ready to fire on thetarget. In embodiments, other sensing inputs and/or sensing devices,user action capture inputs and/or devices, and the like, as describedherein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using sensing inputs/sensing devices and user movements or actionsfor controlling or initiating commands, such as with an optical sensorand an eye shut, blink, and the like movement. The state of thesoldier's eye may be sensed by an optical sensor that is included in theoptical chain of the eyepiece, such as for using eye movement forcontrol of the eyepiece. For instance, the soldier may be aiming theirrifle, where the rifle has the capability to be fired through controlcommands from the eyepiece (such as in the case of a sniper, wherecommanding through the eyepiece may decrease the errors in targeting dueto pulling the trigger manually). The soldier may then fire the weaponthrough a command initiated by the optical sensor detecting apredetermined eye movement, such as in a command profile kept on theeyepiece. In embodiments, other sensing inputs and/or sensing devices,user movements or actions for controlling or initiating commands, andthe like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using sensing inputs/sensing devices and command/control modes andinterfaces in which the inputs can be reflected, such as with aproximity sensor and robotic control interface. A proximity sensorintegrated into the eyepiece may be used to sense the soldier'sproximity to a robotic control interface in order to activate and enablethe use of the robotics. For instance, a soldier walks up to abomb-detecting robot, and the robot automatically activates andinitializes configuration for this particular soldier (e.g. configuringfor the preferences of the soldier). In embodiments, other sensinginputs and/or sensing devices, command and/or control modes andinterfaces in which the inputs can be reflected, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using sensing inputs/sensing devices and applications on platformthat can use commands/respond to inputs, such as with an audio sensorand music/sound application. An audio sensor may monitor the ambientsound and initiate and/or adjust the volume for music, ambient sound,sound cancelling, and the like, to help counter an undesirable ambientsound. For instance, a soldier is loaded onto a transport and theengines of the transport are initially off. At this time the soldier mayhave no other duties except to rest, so they initiate music to help themrest. When the engines of the transport come on the music/soundapplication adjusts the volume and/or initiates additional soundcancelling audio in order to help keep the music input the same asbefore the engines started up. In embodiments, other sensing inputsand/or sensing devices, applications on platform that can use commandsand/or respond to inputs, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using sensing inputs/sensing devices and communication or connectionfrom the on-platform interface to external systems and devices, such aswith a passive IR proximity sensor and external digital signalprocessor. A soldier may be monitoring a night scene with the passive IRproximity sensor, the sensor indicates a motion, and the eyepieceinitiates a connection to an external digital signal processor foraiding in identifying the target from the proximity sensor data.Further, an IR imaging camera may be initiated to contribute additionaldata to the digital signal processor. In embodiments, other sensinginputs and/or sensing devices, communication or connection from theon-platform interface to external systems and devices, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using sensing inputs/sensing devices and useful external devices tobe controlled, such as with an acoustic sensor and a weapons system,where an eyepiece being worn by a soldier senses a loud sound, such asmay be an explosion or gun fire, and where the eyepiece then initiatesthe control of a weapons system for possible action against a targetassociated with the creation of the loud sound. For instance, a soldieris on guard duty, and gunfire is heard. The eyepiece may be able todetect the direction of the gunshot, and direct the soldier to theposition from which the gunshot was made. In embodiments, other sensinginputs and/or sensing devices, useful external devices to be controlled,and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using sensing inputs/sensing devices and applications for thoseuseful external devices, such as with a camera and external applicationfor instructions. The camera embedded in a soldier's eyepiece may view atarget icon indicating that instructions are available, and the eyepieceaccessing the external application for instructions. For instance, asoldier is delivered to a staging area, and upon entry the eyepiececamera views the icon, accesses the instructions externally, andprovides the soldier with the instructions for what to do, where all thesteps may be automatic so that the instructions are provided without thesoldier being aware of the icon. In embodiments, other sensing inputsand/or sensing devices, applications for external devices, and the like,as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using sensing inputs/sensing devices and feedback to user related tothe external devices and applications, such as with a GPS sensor and avisual display from a remote application. The soldier may have anembedded GPS sensor that sends/streams location coordinates to a remotelocation facility/application that sends/streams a visual display of thesurrounding physical environment to the eyepiece for display. Forinstance, a soldier may be constantly viewing the surroundingenvironment though the eyepiece, and by way of the embedded GPS sensor,is continuously streamed a visual display overlay that allows for thesoldier to have an augmented reality view of the surroundingenvironment, even as the change locations. In embodiments, other sensinginputs and/or sensing devices, feedback related to external devicesand/or external applications, and the like, as described herein, mayalso be applied.

In an example, control aspects of the eyepiece may include combinationsof using user action capture inputs/devices and user movements oractions for controlling or initiating commands, such as with a bodymovement sensor (e.g. kinetic sensor) and an arm motion. The soldier mayhave body movement sensors attached to their arms, where the motion oftheir arms convey a command. For instance, a soldier may have kineticsensors on their arms, and the motion of their arms are duplicated in anaircraft landing lighting system, such that the lights normally held bypersonnel aiding in a landing may be made to be larger and more visible.In embodiments, other user action capture inputs and/or devices, usermovements or actions for controlling or initiating commands, and thelike, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using user action capture inputs/devices and command/control modesand interfaces in which the inputs can be reflected, such as wearablesensor sets and a predictive learning-based user interface. A soldiermay wear a sensor set where the data from the sensor set is continuouslycollected and fed to a machine-learning facility through alearning-based user interface, where the soldier may be able to accept,reject, modify, and the like, the learning from their motions andbehaviors. For instance, a soldier may perform the same tasks ingenerally the same physical manner every Monday morning, and themachine-learning facility may establish a learned routine that itprovides to the soldier on subsequent Monday mornings, such as areminder to clean certain equipment, fill out certain forms, playcertain music, meet with certain people, and the like. Further, thesoldier may be able to modify the outcome of the learning through directedits to the routine, such as in a learned behavior profile. Inembodiments, other user action capture inputs and/or devices, commandand/or control modes and interfaces in which the inputs can bereflected, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using user action capture inputs/devices and applications on platformthat can use commands/respond to inputs, such as a finger-followingcamera and video application. A soldier may be able to control thedirection that the eyepiece embedded camera is taking video through aresident video application. For instance, a soldier may be viewing abattle scene where they have need to be gazing in one direction, such asbeing watchful for new developments in the engagement, while filming ina different direction, such as the current point of engagement. Inembodiments, other user action capture inputs and/or devices,applications on platform that can use commands and/or respond to inputs,and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using user action capture inputs/devices and communication orconnection from the on-platform interface to external systems anddevices, such as a microphone and voice recognition input plus asteering wheel control interface. The soldier may be able to changeaspects of the handling of a vehicle through voice commands receivedthrough the eyepiece and delivered to a vehicle's steering wheel controlinterface (such as through radio communications between the eyepiece andthe steering wheel control interface). For instance, a soldier isdriving a vehicle on a road, and so the vehicle has certain handlingcapabilities that are ideal for the road. But the vehicle also has othermodes for diving under different conditions, such as off-road, in snow,in mud, in heavy rain, while in pursuit of another vehicle, and thelike. In this instance, the soldier may be able to change the modethrough voice command as the vehicle changes driving conditions. Inembodiments, other user action capture inputs and/or devices,communication or connection from the on-platform interface to externalsystems and devices, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using user action capture inputs/devices and useful external devicesto be controlled, such as a microphone and voice recognition input plusan automotive dashboard interface device. The soldier may use voicecommands to control various devices associated with the dashboard of avehicle, such as heating and ventilation, radio, music, lighting, tripcomputer, and the like. For instance, a soldier may be driving a vehicleon a mission, across rough terrain, such that they cannot let go of thesteering wheel with either hand in order to manually control a vehicledashboard device. In this instance, the soldier may be able to controlthe vehicle dashboard device through voice controls to the eyepiece.Voice commands through the eyepiece may be especially advantageous, suchas opposed to voice control through a dashboard microphone system,because the military vehicle may be immersed in a very loud acousticenvironment, and so using the microphone in the eyepiece may givesubstantially improved performance under such conditions. Inembodiments, other user action capture inputs and/or devices, usefulexternal devices to be controlled, and the like, as described herein,may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using user action capture inputs/devices and applications for usefulexternal devices, such as with a joystick device and externalentertainment application. A soldier may have access to a gamingjoystick controller and is able to play a game through an externalentertainment application, such as a multi-player game hosted on anetwork server. For instance, the soldier may be experiencing down timeduring a deployment, and on base they have access to a joystick devicethat interfaces to the eyepiece, and the eyepiece in turn to theexternal entertainment application. In embodiments, the soldier may benetworked together with other military personnel across the network. Thesoldier may have stored preferences, a profile, and the like, associatedwith the game play. The external entertainment application may managethe game play of the soldier, such as in terms of their deployment,current state of readiness, required state of readiness, past history,ability level, command position, rank, geographic location, futuredeployment, and the like. In embodiments, other user action captureinputs and/or devices, applications for external devices, and the like,as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using user action capture inputs/devices and feedback to the userrelated to external devices and applications, such as with an activitydetermination system and tonal output or sound warning. The soldier mayhave access to the activity determination system through the eyepiece tomonitor and determine the soldier's state of activity, such as inextreme activity, at rest, bored, anxious, in exercise, and the like,and where the eyepiece may provide forms of tonal output or soundwarning when conditions go out of limits in any way, such as pre-set,learned, as typical, and the like. For instance, the soldier may bemonitored for current state of health during combat, and where thesoldier and/or another individual (e.g. medic, hospital personnel,another member of the soldier's team, a command center, and the like)are provided an audible signal when health conditions enter a dangerouslevel, such as indicating that the soldier has been hurt in battle. Assuch, others may be alerted to the soldier's injuries, and would be ableto attend to the injuries in a more time effective manner. Inembodiments, other user action capture inputs and/or devices, feedbackrelated to external devices and/or external applications, and the like,as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using user movements or actions for controlling or initiatingcommands plus command/control modes and interfaces in which the inputscan be reflected, such as a clenched fist and Navigable list. A soldiermay bring up a navigable list as projected content on the eyepiecedisplay with a gesture such as a clenched fist, and the like. Forinstance, the eyepiece camera may be able to view the soldier's handgesture(s), recognize and identify the hand gesture(s), and execute thecommand in terms of a pre-determined gesture-to-command database. Inembodiments, hand gestures may include gestures of the hand, finger,arm, leg, and the like. In embodiments, other user movements or actionsfor controlling or initiating commands, command and/or control modes andinterfaces in which the inputs can be reflected, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using user movements or actions for controlling or initiatingcommands plus applications on platform that can use commands/respond toinputs, such as a head nod and information display. The soldier maybring up an information display application with a gesture such as aheadshake, arm motion, leg motion, eye motion, and the like. Forinstance, the soldier may wish to access an application, database,network connection, and the like, through the eyepiece, and is able tobring up a display application as part of a graphical user interfacewith the nod of their head (such as sensed though motion detectors inthe eyepiece, on the soldier's head, on the soldier's helmet, and thelike. In embodiments, other user movements or actions for controlling orinitiating commands, applications on platform that can use commandsand/or respond to inputs, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using user movements or actions for controlling or initiatingcommands plus communication or connection from the on-platform interfaceto external systems and devices, such as the blink of an eye and throughan API to external applications. The soldier may be able to bring up anapplication program interface to access external applications, such aswith the blink of an eye, a nod of the head, the movement of an arm orleg, and the like. For instance, the soldier may be able to access anexternal application through an API embedded in an eyepiece facility,and do so with the blink of an eye, such as detected though an opticalmonitoring capability through the optics system of the eyepiece. Inembodiments, other user movements or actions for controlling orinitiating commands, communication or connection from the on-platforminterface to external systems and devices, and the like, as describedherein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using user movements or actions for controlling or initiatingcommands and external devices to be controlled, such as through the tapof a foot accessing an external range finder device. A soldier may havea sensor such as a kinetic sensor on their shoe that will detect themotion of the soldier's foot, and the soldier uses a foot motion such asa tap of their foot to use an external range finder device to determinethe range to an object such an enemy target. For instance, the soldiermay be targeting a weapon system, and using both hands in the process.In this instance, commanding by way of a foot action through theeyepiece may allow for ‘hands free’ commanding. In embodiments, otheruser movements or actions for controlling or initiating commands, usefulexternal devices to be controlled, and the like, as described herein,may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using user movements or actions for controlling or initiatingcommands plus applications for those useful external devices, such asmaking a symbol with a hand and an information conveying application.The soldier may utilize a hand formed symbol to trigger informationshared through an external information conveying application, such as anexternal information feed, a photo/video sharing application, a textapplication, and the like. For instance, a soldier uses a hand signal toturn on the embedded camera and share the video stream with anotherperson, to storage, and the like. In embodiments, other user movementsor actions for controlling or initiating commands, applications forexternal devices, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using user movements or actions for controlling or initiatingcommands plus feedback to soldier as related to an external device andapplication, such as a headshake plus an audible alert. The soldier maybe wearing an eyepiece equipped with an accelerometer (or like capablesensor for detecting g-force headshake), where when the soldierexperiences a g-force headshake that is at a dangerously high level, anaudible alert is sounded as feedback to the user, such as determinedeither as a part of on- or off-eyepiece applications. Further, theoutput of the accelerometer may be recorded and stored for analysis. Forinstance, the soldier may experience a g-force headshake from aproximate explosion, and the eyepiece may sense and record the sensordata associated with the headshake. Further, headshakes of a dangerouslevel may trigger automatic actions by the eyepiece, such astransmitting an alert to other soldiers and/or to a command center,begin monitoring and/or transmitting the health of the soldier fromother body mounted sensors, provide audible instructions to the soldierrelated to their potential injuries, and the like. In embodiments, otheruser movements or actions for controlling or initiating commands,feedback related to external devices and/or external applications, andthe like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using command/control modes and interfaces in which the inputs can bereflected plus applications on platform that can use commands/respond toinputs, such as a graphical user interface plus various applicationsresident on the eyepiece. The eyepiece may provide a graphical userinterface to the soldier and applications presented for selection. Forinstance, the soldier may have a graphical user interface projected bythe eyepiece that provides different domains of application, such asmilitary, personal, civil, and the like. In embodiments, other commandand/or control modes and interfaces in which the inputs can bereflected, applications on platform that can use commands and/or respondto inputs, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using command/control modes and interfaces in which the inputs can bereflected plus a communication or connection from the on-platforminterface to external systems and devices, such as a 3D navigationeyepiece interface plus navigation system controller interface toexternal system. The eyepiece may enter a navigation mode and connect toan external system through a navigation system controller interface. Forinstance, a soldier is in military maneuvers and brings up a preloaded3D image of the surrounding terrain through the eyepiece navigationmode, and the eyepiece automatically connects to the external system forupdates, current objects of interest such as overlaid by satelliteimages, and the like. In embodiments, other command and/or control modesand interfaces in which the inputs can be reflected, communication orconnection from the on-platform interface to external systems anddevices, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using command/control modes and interfaces in which the inputs can bereflected plus an external device to be controlled, such as an augmentedreality interface plus external tracking device. The soldier's eyepiecemay enter into an augmented reality mode and interface with an externaltracking device to overlay information pertaining to the location of atraced object or person with an augmented reality display. For instance,the augmented reality mode may include a 3D map, and a person's locationas determined by the external tracking device may be overlaid onto themap, and show a trail as the tracked person moves. In embodiments, othercommand and/or control modes and interfaces in which the inputs can bereflected, useful external devices to be controlled, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using command/control modes and interfaces in which the inputs can bereflected plus applications for those external devices, such assemi-opaque display mode plus simulation application. The eyepiece maybe placed into a semi-opaque display mode to enhance the display of asimulation display application to the solder. For instance, the soldieris preparing for a mission, and before entering the field the soldier isprovided a simulation of the mission environment, and since there is noreal need for the user to see the real environment around them duringthe simulation, the eyepiece places the eyepiece into a semi-opaquedisplay mode. In embodiments, other command and/or control modes andinterfaces in which the inputs can be reflected, applications forexternal devices, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using command/control modes and interfaces in which the inputs can bereflected plus feedback to user related to the external devices andapplications, such as an auditory command interface plus a tonal outputfeedback. The soldier may place the eyepiece into an auditory commandinterface mode and the eyepiece responds back with a tonal output asfeedback from the system that the eyepiece is ready to receive theauditory commands. For instance, the auditory command interface mayinclude at least portions of the auditory command interface in anexternal location, such as out on a network, and the tone is providedonce the entire system is ready to accept auditory commands. Inembodiments, other command and/or control modes and interfaces in whichthe inputs can be reflected, feedback related to external devices and/orexternal applications, and the like, as described herein, may also beapplied.

In an example, control aspects of the eyepiece may include combinationsof using applications on platform that can use commands/respond toinputs plus Communication or connection from the on-platform interfaceto external systems and devices, such as a communication applicationplus a network router, where the soldier is able to open up acommunications application, and the eyepiece automatically searches fora network router for connectivity to a network utility. For instance, asoldier is in the field with their unit, and a new base camp isestablished. The soldier's eyepiece may be able to connect into thesecure wireless connection once communications facilities have beenestablished. Further, the eyepiece may alert the soldier oncecommunications facilities have been established, even if the soldier hasnot yet attempted communications. In embodiments, other applications onplatform that can use commands and/or respond to inputs, communicationor connection from the on-platform interface to external systems anddevices, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using applications on platform that can use commands/respond toinputs plus useful external devices to be controlled, such as a videoapplication plus and external camera. The soldier may interface withdeployed cameras, such as for surveillance in the field. For instance,mobile deployable cameras may be dropped from an aircraft, and thesoldier then has connection to the cameras through the eyepiece videoapplication. In embodiments, other applications on platform that can usecommands and/or respond to inputs, useful external devices to becontrolled, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using applications on platform that can use commands/respond toinputs plus applications for external devices, such as an on-eyepiecesearch application plus an external search application. A searchapplication on the eyepiece may be augmented with an external searchapplication. For instance, a soldier may be searching for the identityof an individual that is being questioned, and when the on-eyepiecesearch results in no find, the eyepiece connects with an external searchfacility. In embodiments, other applications on platform that can usecommands and/or respond to inputs, applications for external devices,and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using applications on platform that can use commands/respond toinputs plus feedback to the soldier as related to the external devicesand applications, such as an entertainment application plus aperformance indicator feedback. The entertainment application may beused as a resting mechanism for a soldier that needs to rest but may beotherwise anxious, and performance feedback is designed for the soldierin given environments, such as in a deployment when they need to restbut remain sharp, during down time when attentiveness is declining andneeds to be brought back up, and the like. For instance, a soldier maybe on a transport and about to enter an engagement. In this instance, anentertainment application may be an action-thinking game to heightenattention and aggressiveness, and where the performance indicatorfeedback is designed to maximize the soldier's desire to perform and tothink through problems in a quick and efficient manner. In embodiments,other applications on platform that can use commands and/or respond toinputs, feedback related to external devices and/or externalapplications, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using a communication or connection from the on-platform interface toexternal systems and devices plus external devices to be controlled,such as an on-eyepiece processor interface to external facilities plusan external projector. The eyepiece processor may be able to connect toan external projector so that others may view the content available tothe eyepiece. For instance, a soldier may be in the field and has accessto content that they need to share with others who are not wearing aneyepiece, such as individuals not in the military. In this instance, thesoldier's eyepiece may be able to interface with an external projector,and feed content from the eyepiece to the projector. In embodiments, theprojector may be a pocket projector, a projector in a vehicle, in aconference room, remotely located, and the like. In embodiments theprojector may also be integrated into the eyepiece, such that thecontent may be externally projected from the integrated projector. Inembodiments, other communication or connection from the on-platforminterface to external systems and devices, useful external devices to becontrolled, and the like, as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using a communication or connection from the on-platform interface toexternal systems and devices plus an application for external devices,such as an audio system controller interface plus an external soundsystem. The soldier may be able to connect the audio portion of theeyepiece facilities (e.g. music, audio playback, audio network files,and the like) to an external sound system. For instance, the soldier maybe able to patch a communications being received by the eyepiece to avehicle sound system so that others can hear. In embodiments, othercommunication or connection from the on-platform interface to externalsystems and devices, applications for external devices, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using a communication or connection from the on-platform interface toexternal systems and devices plus feedback to a soldier related to theexternal devices and applications, such as a stepper controllerinterface plus status feedback. The soldier may have access and controlof a mechanism with digital stepper control through a stepper controllerinterface, where the mechanism provides feedback to the soldier as tothe state of the mechanism. For instance, a solder working on removing aroadblock may have a lift mechanism on their vehicle, and the soldiermay be able to directly interface with the lift mechanism through theeyepiece. In embodiments, other communication or connection from theon-platform interface to external systems and devices, feedback relatedto external devices and/or external applications, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using external devices to be controlled plus applications for thoseexternal devices, such as storage-enabled devices plus automatic backupapplications. The soldier in the field may be provided data storagefacilities and associated automatic backup applications. For instance,the storage facility may be located in a military vehicle, so that datamay be backed up from a plurality of soldier's eyepieces to the vehicle,especially when a network link is not available to download to a remotebackup site. A storage facility may be associated with an encampment,with a subset of soldiers in the field (e.g. in a pack), located on thesoldier themselves, and the like. In embodiments, a local storagefacility may upload the backup when network service connections becomeavailable. In embodiments, other useful external devices to becontrolled, applications for external devices, and the like, asdescribed herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using external devices to be controlled plus feedback to a soldierrelated to external devices and applications, such as an externalpayment system plus feedback from the system. The soldier may haveaccess to a military managed payment system, and where that systemprovides feedback to the soldier (e.g. receipts, account balance,account activity, and the like). For instance, the soldier may makepayments to a vendor via the eyepiece where the eyepiece and externalpayment system exchange data, authorization, funds, and the like, andthe payment system provides feedback data to the soldier. Inembodiments, other useful external devices to be controlled, feedbackrelated to external devices and/or external applications, and the like,as described herein, may also be applied.

In an example, control aspects of the eyepiece may include combinationsof using applications for external devices plus feedback to a soldierrelated to external devices and applications, such as an informationdisplay from an external 3D mapping-rendering facility plus feedbackalong with the information display. The soldier may be able to have 3Dmapping information data displayed through the eyepiece, where themapping facility may provide feedback to the soldier, such as based onpast information delivered, past information requested, requests fromothers in the area, based on changes associated with the geographicalarea, and the like. For instance, a soldier may be receiving a 3D maprendering from an external application, where the external applicationis also providing 3D map rendering to at least a second soldier in thesame geographic area. The soldier may then receive feedback from theexternal facility related to the second soldier, such as their positiondepicted on the 3D map rendering, identity information, history ofmovement, and the like. In embodiments, other applications for externaldevices, feedback related to external devices and/or externalapplications, and the like, as described herein, may also be applied.

In embodiments, the eyepiece may provide a user with various forms ofguidance in responding to medical situations. As a first example, theuser may use the eyepiece for training purposes to simulate medicalsituations that may arise in combat, training, on or off duty and thelike. The simulation may be geared towards a medical professional ornon-medical personnel.

By way of example, a low level combat soldier may use the eyepiece toview a medical simulation as part of a training module to providetraining for response to medical situations on the battlefield. Theeyepiece may provide an augmented environment where the user viewsinjuries overlaid on another solider to simulate those common or capableof being found on the battlefield. The soldier may then be promptedthrough a user interface to respond to the situation as presented. Theuser may be given step-by-step instructions of a course of action inproviding emergency medical care on the field, or the user may carry outactions in response to the situation that are then corrected until theappropriate response is given.

Similarly, the eyepiece may provide a training environment for a medicalprofessional. The eyepiece may present the user with a medical emergencyor situation requiring a medical response for the purpose of trainingthe medical professional. The eyepiece may play out common battle fieldscenarios for which the user must master appropriate responses andlifesaving techniques.

By way of example, the user may be presented with an augmented realityof a wounded soldier with a gunshot wound to the soldier's body. Themedical professional may then act out the steps he feels to be theappropriate response for the situation, select steps through a userinterface of the eyepiece that he feels are appropriate for thesituation, input the steps into a user interface of the eyepiece, andthe like. The user may act out the response through use of sensors andor an input device or he may input the steps of his response into a userinterface via eye movements, hand gestures and the like. Similarly, hemay select the appropriate steps as presented to him through the userinterface via eye movements, hand gestures and the like. As actions arecarried out and the user makes decisions about treatment, the user maybe presented with additional guidance and instruction based on hisperformance. For example, if the user is presented with a soldier with agunshot wound to the chest, and the user begins to lift the soldier to adangerous position, the user may be given a warning or prompt to changehis course of treatment. Alternatively, the user may be prompted withthe correct steps in order to practice proper procedure. Further, thetrainee may be presented with an example of a medical chart for thewounded soldier in the training situation where the user may have tobase his decisions at least in part on what is contained in the medicalchart. In various embodiments, the user's actions and performance may berecorded and or documented by the eyepiece for further critiquing andinstruction after the training session has paused or otherwise stopped.

In embodiments, the eyepiece may provide a user with various forms ofguidance in responding to actual medical situations in combat. By way ofexample, a non-trained soldier may be prompted with step-by-step lifesaving instructions for fellow soldiers in medical emergencies when amedic is not immediately present. When a fellow soldier is wounded, theuser may input the type of injury, the eyepiece may detect the injury ora combination of these may occur. From there, the user may be providedwith life saving instruction with which to treat the wounded soldier.Such instruction may be presented in the form of augmented reality in astep-wise process of instructions for the user. Further, the eyepiecemay provide augmented visual aids to the user regarding location ofvital organs near the wounded soldier's injury, an anatomical overlay ofthe soldier's body and the like. Further, the eyepiece may take video ofthe situation that is then sent back to a medic not in the field or onhis way to the field, thereby allowing the medic to walk the untraineduser through an appropriate lifesaving technique on the battlefield.Further, the wounded soldier's eyepiece may send vital information, suchas information collected through integral or associated sensors, aboutthe wounded soldier to the treating soldier's eyepiece to be sent to themedic or it may be sent directly to the medic in a remote location suchthat the treating soldier may provide the wounded solider with medicalhelp based on the information gathered from the wounded soldier'seyepiece.

In other embodiments, when presented with a medical emergency on thebattlefield, a trained medic may use the eyepiece to provide ananatomical overlay of the soldier's body so that he may respond moreappropriately to the situation at hand. By way of example only and notto limit the present invention, if the wounded soldier is bleeding froma gunshot wound to the leg, the user may be presented with an augmentedreality view of the soldier's arteries such that the user may determinewhether an artery has been hit and how severe the wound may be. The usermay be presented with the proper protocol via the eyepiece for the givenwound so that he may check each step as he moves through treatment. Suchprotocol may also be presented to the user in an augmented reality,video, audio or other format. The eyepiece may provide the medic withprotocols in the form of augmented reality instructions in a step-wiseprocess. In embodiments, the user may also be presented with anaugmented reality overlay of the wounded soldier's organs in order toguide the medic through any procedure such that the medic does not doadditional harm to the soldier's organs during treatment. Further, theeyepiece may provide augmented visual aids to the user regardinglocation of vital organs near the wounded soldier's injury, ananatomical overlay of the soldier's body and the like.

In embodiments, the eyepiece may be used to scan the retina of thewounded soldier in order to pull up his medical chart on thebattlefield. This may alert the medic to possible allergies tomedication or other important issues that may provide a benefit duringmedical treatment.

Further, if the wounded soldier is wearing the eyepiece, the device maysend information to the medic's glasses including the wounded soldier'sheart rate, blood pressure, breathing stress, and the like. The eyepiecemay also help the user observe the walking gait of a soldier todetermine if the soldier has a head injury and they may help the userdetermine the location of bleeding or an injury. Such information mayprovide the user with information of possible medical treatment, and inembodiments, the proper protocol or a selection of protocols may bedisplayed to the user to help him in treating the patient.

In other embodiments, the eyepiece may allow the user to monitor othersymptoms of the patient for a mental health status check. Similarly, theuser can check to determine if the patient is exhibiting rapid eyemovement and further may use the eyepiece to provide the patient withcalming treatment such as providing the patient with eye movementexercises, breathing exercises, and the like. Further, the medic may beprovided with information regarding the wounded soldier's vital signsand health data as it is collected from the wounded soldier's eyepieceand sent to the medic's eyepiece. This may provide the medic with realtime data from the wounded soldier without having to determine such dataon his own for example by taking the wounded soldier's blood pressure.

In various embodiments, the user may be provided with alerts from theeyepiece that tells him how for away an air or ground rescue is from hislocation on the battlefield. This may provide a medic with importantinformation and alert him to whether certain procedures should or mustbe attempted given the time available in the situation, and it mayprovide an injured soldier with comfort knowing help is on the way oralert him that he may need other sources of help.

In other embodiments, the user may be provided alerts of his own vitalsigns if a problem is detected. For example, a soldier may be alerted ifhis blood pressure is too high, thereby alerting him that he must takemedication or remove himself from combat if possible to return his bloodpressure to a safe level. Also, the user may be alerted of other suchpersonal data such as his pupil size, heart rate, waking gait change andthe like in order to determine if the user is experiencing a medicalproblem. In other embodiments, a user's eyepiece may also alert medicalpersonnel in another location of the user's medical status in order tosend help for the user whether or not he knows he requires such help.Further, general data may be aggregated from multiple eyepieces in orderto provide the commanding office with detailed information on hiswounded soldiers, how many soldiers he has in combat, how many of thoseare wounded, and the like.

In various embodiments, a trained medical professional may use theeyepiece in medical responses out of combat as well. Such eyepiece mayhave similar uses as described above on or off the home base of themedic but outside of combat situations. In this way, the eyepiece mayprovide a user with a means to gain augmented reality assistance duringa medical procedure, to document a medical procedure, perform a medicalprocedure at the guidance of a remote commanding officer via videoand/or audio, and the like on or off a military base. This may provideassistance in a plurality of situations where the medic may needadditional assistance. An example of this may occur when the medic is onduty on a training exercise, a calisthenics outing, a military hike andthe like. Such assistance may be of importance when the medic is theonly responder, when he is a new medic, approached with a new situationand the like.

In some embodiments, the eyepiece may provide user guidance in anenvironment related to a military transport plane. For example, theeyepiece may be used in such an environment when training, going intobattle, on a reconnaissance or rescue mission, while moving equipment,performing maintenance on the plane and the like. Such use may be suitedfor personnel of various ranks and levels.

For illustrative purposes, a user may receive audio and visualinformation through the eyepiece while on the transport plane and goinginto a training exercise. The information may provide the user withdetails about the training mission such as the battle field conditions,weather conditions, mission instructions, map of the area and the like.The eyepiece may simulate actual battle scenarios to prepare the userfor battle. The eyepiece may also record the user's responses andactions through various means. Such data gathering may allow the user toreceive feedback about his performance. Further, the eyepiece may thenchange the simulation based on the results obtained during the trainingexercise to change the simulation while it is underway or to changefuture simulations for the user or various users.

In embodiments, the eyepiece may provide user guidance and orinteraction on a military transport plane when going into battle. Theuser may receive audio and visual information about the mission as theuser boards the plane. Check lists may be presented to the user forensuring he has the appropriate materials and equipment of the mission.Further, instructions for securing equipment and proper use of safetyharnesses may be presented along with information about the aircraftsuch as emergency exits, location of oxygen tanks, and safety devices.The user may be presented with instructions such as when to rest priorto the mission and have a drug administered for that purpose. Theeyepiece may provide the user with noise cancellation for rest prior tomission, and then may alert the user when his rest is over and furthermission preparation is to begin. Additional information may be providedsuch as a map of the battle area, number of vehicles and/or people onthe field, weather conditions of the battle area and the like. Thedevice may provide a link to other soldiers so that instructions andbattle preparation may include soldier interaction where the commandingofficer is heard by subordinates and the like. Further, information foreach user may be formatted to suit his particular needs. For example, acommanding officer may receive higher level or more confidentialinformation that may not be necessary to provide a lower rankingofficer.

In embodiments, the user may use the eyepiece on a military transportplane in a reconnaissance or rescue mission where the eyepiece capturesand stores various images and or video of places of interest as it fliesover areas which may be used for gaining information about a potentialground battle area and the like. The eyepiece may be used to detectmovement of people and vehicles on the ground and thereby detect enemyto be defeated or friendlies to be rescued or assisted. The eyepiece mayprovide the ability to apply tags to a map or images of areas flown overand searched giving a particular color coding for areas that have beensearched or still need to be searched.

In embodiments, a user on a military transport plane may be providedwith instructions and or a checklist for equipment to be stocked, thequantity and location to be moved and special handling instructions forvarious equipment. Alerts may be provided to the user for approachingvehicles as items are unloaded or loaded in order to ensure security.

For maintenance and safety of the military transport plane, the user maybe provided with a preflight check for proper functioning of theaircraft. The pilot may be alerted if proper maintenance was notcompleted prior to mission. Further, the aircraft operators may beprovided with a graphic overview or a list of the aircraft history totrack the history of the aircraft maintenance.

In some embodiments, the eyepiece may provide user guidance in anenvironment related to a military fighter plane. For example, theeyepiece may be used in such an environment when training, going intobattle, for maintenance and the like. Such use may be suited forpersonnel of various ranks and levels.

By way of example, a user may use the eyepiece for training for militaryfighter plane combat. The user may be presented with augmented realitysituations that simulate combat situations in a particular military jetor plane. The user's responses and actions may be recorded and oranalyzed to provide the user with additional information, critique andto alter training exercises based on past data.

In embodiments related to actual combat, the user may be presented withinformation showing him friendly and non-friendly aircraft surroundingand/or approaching him. The user may be presented information regardingthe enemy aircraft such as top speed, maneuvering ability and missilerange. In embodiments, the user may receive information relating to thepresence of ground threats and may be alerted about the same. Theeyepiece may sync to the user's aircraft and or aircraft instruments andgauges such that the pilot may see emergency alerts and additionalinformation regarding the aircraft that may not normally be displayed inthe cockpit. Further, the eyepiece may display the number of seconds totargeted area, the time to fire a missile or eject from the aircraftbased on incoming threats. The eyepiece may suggest maneuvers for thepilot to preform based on the surrounding environment, potential threatsand the like. In embodiments, the eyepiece may detect and displayfriendly aircraft even when such aircraft is in stealth mode.

In embodiments, the user may be provided with a preflight check forproper functioning of the fighter aircraft. The pilot may be alerted ifproper routing maintenance was not completed prior to mission by linkingwith maintenance records, aircraft computers and otherwise. The eyepiecemay allow the pilot to view history of the aircraft maintenance alongwith diagrams and schematics of the same.

In some embodiments, the eyepiece may provide user guidance in anenvironment related to a military helicopter. For example, the eyepiecemay be used in such an environment when training, going into combat, formaintenance and the like. Such use may be suited for personnel ofvarious ranks and levels.

By way of example, a user may use the eyepiece for training for militaryhelicopter operation in combat or high stress situation. The user may bepresented with augmented reality situations that simulate combatsituations in a particular aircraft. The user's responses and actionsmay be recorded and or analyzed to provide the user with additionalinformation, critique and to alter training exercises based on pastdata.

During training and/or combat a user's eyepiece may sync into theaircraft for alerts about the vital statistics and maintenance of theaircraft. The user may view program and safety procedures and emergencyprocedures for passengers as he boards the aircraft. Such procedures mayshow how to ride in the aircraft safely, how to operate the doors forentering and exiting the aircraft, the location of lifesaving equipment,among other information. In embodiments, the eyepiece may present theuser with the location and/or position of threats such as those thatcould pose a danger to a helicopter during its typical flight. Forexample, the user may be presented with the location of low flyingthreats such as drones, other helicopters and the location of landthreats. In embodiments, noise cancelling earphones and a multi-useruser interface may be provided with the eyepiece allowing forcommunication during flight. In an event where the helicopter goes down,the user's eyepiece may transmit the location and helicopter informationto a commanding officer and a rescue team. Further, use of night visionof the eyepiece during a low flying mission may enable a user to turn ahigh-powered helicopter spotlight off in order to search or find enemywithout being detected.

In embodiments, and as described in various instances herein, theeyepiece may provide assistance in tracking the maintenance of theaircraft and to determine if proper routine maintenance has beenperformed. Further, and with other aircraft and vehicles mentionedherein, augmented reality may be used in the assistance of maintainingand working on the aircraft.

In some embodiments, the eyepiece may provide user guidance in anenvironment related to a military drone aircraft or robots. For example,the eyepiece may be used in such an environment in reconnaissance,capture and rescue missions, combat, in areas that pose particulardanger to humans, and the like.

In embodiments, the eyepiece may provide video feed to the userregarding the drone's surrounding environment. Real time video may bedisplayed for up to the second information about various areas ofinterest. Gathering such information may provide a soldier with theknowledge of the number of enemy soldiers in the area, the layout ofbuildings and the like. Further, data may be gathered and sent to theeyepiece from the drone and or robot in order to gather intelligence onthe location of persons of interest to be captured or rescued. By way ofillustration, a user outside of a secure compound or bunker may use thedrone and or robot to send back video or data feed to of the location,number and activity of persons in the secure compound in preparation ofa capture or rescue.

In embodiments, use of the eyepiece with a drone and/or robot may allowa commanding officer to gather battlefield data during a mission to makeplan changes and to give various instructions of the team depending onthe data gathered. Further, the eyepiece and controls associatedtherewith may allow users to deploy weapons on the drone and/or robotvia a user interface in the eyepiece. The data feed sent from the droneand/or robot may give the user information as to what weapons to deployand when to deploy them.

In embodiments, the data gathered from the drone and/or robot may allowthe user to get up close to potential hazardous situations. For examplethis may allow the user to investigate biological spills, bombs,alleyways, foxholes, and the like to provide the user with data of thesituation and environment while keeping him out of direct harm's way.

In some embodiments, the eyepiece may provide user guidance in anenvironment related to a military ship at sea. For example, the eyepiecemay be used in such an environment when training, going into battle,performing a search and rescue mission, performing disaster clean up,when performing maintenance and the like. Such use may be suited forpersonnel of various ranks and levels.

In embodiments, the eyepiece may be used in training to prepare users ofvarious skill sets for performance of their job duties on the vessel.The training may include simulations testing the user's ability tonavigate, control the ship and/or perform various tasks while in acombat situation, and the like. The user's responses and actions may berecorded and or analyzed to provide the user with additionalinformation, critique and to alter training exercises based on pastdata.

In embodiments, the eyepiece may allow the user to view potential shipthreats out on the horizon by providing him with an augmented realityview of the same. Such threats may be indicated by dots, graphics, orother means. Instructions may be sent to the user via the eyepieceregarding preparation for enemy engagement once the eyepiece detects aparticular threat. Further, the user may view a map or video of the portwhere they will dock and be provided with enemy location. Inembodiments, the eyepiece may allow the user to sync with the shipand/or weapon equipment to guide the user in the use of the equipmentduring battle. The user may be alerted by the eyepiece to whereinternational and national water boundaries lie.

In embodiments where search and rescue is needed, the eyepiece mayprovide for tracking the current and/or for tagging the area of waterrecently searched. In embodiments where the current is tracked, this mayprovide the user information conveying the potential location or changedlocation of persons of interest to be rescued. Similarly, the eyepiecemay be used in environments where the user must survey the surroundingenvironment. For example, the user may be alerted to significant shiftsin water pressure and/or movement that may signal mantle movement and orthe imminence of an upcoming disaster. Alerts may be sent to the uservia the eyepiece regarding the shifting of the mantle, threat ofearthquake and/or tsunami and the like. Such alerts may be provided bythe eyepiece synching with devices on the ship, by tracking ocean watermovement, current change, change in water pressure, a drop or increaseof the surrounding water and the like.

In embodiments where military ships are deployed for disaster clean up,the eyepiece may be used in detecting areas of pollution, the speed oftravel of the pollution and predictions of the depth and where thepollution will settle. In embodiments the eyepiece may be useful indetecting the parts per million of pollution and the variance thereon todetermine the change in position of the volume of the pollution.

In various embodiments the eyepiece may provide a user with a program tocheck for proper functioning of the ship and the equipment thereon.Further, various operators of the ship may be alerted if proper routinemaintenance was not completed prior to deployment. In embodiments theuser may also be able to view the maintenance history of the ship alongwith the status of vital functioning of the ship.

In embodiments, the eyepiece may provide a user with various forms ofguidance in the environment of a submarine. For example, the eyepiecemay be used in such an environment when training, going into combat, formaintenance and the like. Such use may be suited for personnel ofvarious ranks and levels.

By way of example, a user may use the eyepiece for training forsubmarine operation in combat or high stress situation. The user may bepresented with augmented reality situations or otherwise that simulatecombat situations in a particular submarine. The training program may bebased on the user's rank such that his rank will determine the type ofsituation presented. The user's responses and actions may be recordedand or analyzed to provide the user with additional information,critique and to alter training exercises based on past data. Inembodiments, the eyepiece may also train the user in maintaining thesubmarine, use of the submarine and proper safety procedures and thelike.

In combat environments, the eyepiece may be used to provide the userwith information relating to the user's depth, the location of the enemyand objects, friendlies and/or enemies on the surface. In embodiments,such information may be conveyed to the user in a visual representation,through audio and the like. In various embodiments the eyepiece may syncinto and/or utilize devices and equipment of the submarine to gatherdata from GPS, sonar and the like to gather various information such asthe location of other objects, submarines, and the like. The eyepiecemay display instructions to the soldier regarding safety procedures,mission specifics, and the presences of enemies in the area. Inembodiments, the device may communicate or sync with the ship and/orweapon equipment to guide the soldier in the use of such equipment andto provide a display relating to the particular equipment. Such displaymay include a visual and audio data relating to the equipment. Byfurther way of example, the device may be used with the periscope toaugment the user's visual picture and/or audio to show potentialthreats, places of interest, and information that may not otherwise bedisplayed by using the periscope such as the location of enemies out ofview, national and international water boundaries, various threats, andthe like.

The eyepiece may also be used in maintenance of the submarine. Forexample, it may provide the user with a pre journey check for properfunctioning of the ship, it may alert the operation of proper routinemaintenance was performed or not completed prior to the mission.Further, a user may be provided with a detailed history to reviewmaintenance performed and the like. In embodiments, the eyepiece mayalso assist in maintaining the submarine by providing an augmentedreality or other program that instructs the user in performing suchmaintenance.

In embodiments, the eyepiece may provide a user with various forms ofguidance in the environment of a ship in port. For example, the eyepiecemay be used in such an environment when training, going into combat, formaintenance and the like. Such use may be suited for personnel ofvarious ranks and levels.

By way of example, a user may use the eyepiece for training for a shipin a port when in combat, under attach or a high stress situation. Theuser may be presented with augmented reality situations, or otherwise,that simulate combat situations that may be seen in a particular portand on such a ship. The training program may show various ports fromaround the world and the surrounding land data, data for the number ofally ships or enemy ships that may be in the port at a give time, and itmay show the local fueling stations and the like. The training programmay be based on the user's rank such that his rank will determine thetype of situation presented. The user's responses and actions may berecorded and/or analyzed to provide the user with additionalinformation, critique and to alter training exercises based on pastdata. In embodiments, the eyepiece may also train the user inmaintaining and performing mechanical maintenance on the ship, use ofthe ship and proper safety procedures to employ on the ship and thelike.

In combat environments, the eyepiece may be used to provide the userwith information relating to the port where the user will or is docked.They user may be provided with information on the location or othervisual representation of the enemy and or friendly ships in the port. Inembodiments, the user may obtain alerts of approaching aircraft andenemy ships and the user may sync into the ship and/or weapon equipmentto guide the user in using the equipment while providing informationand/or display data about the equipment. Such data may include theamount and efficacy of particular ammunition and the like. The eyepiecemay display instructions to the soldier regarding safety procedures,mission specifics, and the presences of enemies in the area. Suchdisplay may include visual and/or audio information.

The eyepiece may also be used in maintenance of the ship. For example,it may provide the user with a pre-journey check for proper functioningof the ship, it may alert the operation of proper routine maintenancewas performed or not completed prior to the mission. Further, a user maybe provided with a detailed history to review maintenance performed andthe like. In embodiments, the eyepiece may also assist in maintainingthe ship by providing an augmented reality or other program thatinstructs the user in performing such maintenance.

In other embodiments, the user may use the eyepiece or other device togain biometric information of those coming into the port. Suchinformation may provide the user's identity and allow the user to knowif the person is a threat or someone of interest. In other embodiments,the user may scan an object or container imported into the port forpotential threats in shipments of cargo and the like. The user may beable to detect hazardous material based on density or various otherinformation collected by the sensors associated with the eyepiece ordevice. The eyepiece may record information or scan a document todetermine whether the document may be counterfeit or altered in someway. This may assist the user in checking an individual's credentials,and it may be used to check the papers associated with particular piecesof cargo to alert the user to potential threats or issues that may berelated to the cargo such as inaccurate manifests, counterfeitdocuments, and the like.

In embodiments, the eyepiece may provide a user with various forms ofguidance when using a tank or other land vehicles. For example, theeyepiece may be used in such an environment when training, going intocombat, for surveillance, group transport, for maintenance and the like.Such use may be suited for personnel of various ranks and levels.

By way of example, a user may use the eyepiece for training for using atank or other ground vehicle when in combat, under attack or a highstress situation or otherwise. The user may be presented with augmentedreality situations, or otherwise, that simulate combat situations thatmay be seen when in and/or operating a tank. The training program maytest the user on proper equipment and weapon use and the like. Thetraining program may be based on the user's rank such that his rank willdetermine the type of situation presented. The user's responses andactions may be recorded and/or analyzed to provide the user withadditional information, critique and to alter training exercises basedon past data. In embodiments, the eyepiece may also train the user inmaintaining the tank, use of the tank and proper safety procedures toemploy when in the tank or land vehicle and the like.

In combat environments, the eyepiece may be used to provide the userwith information and/or visual representations relating to the locationof the enemy and/or friendly vehicles on the landscape. In embodiments,the user may obtain alerts of approaching aircraft and enemy vehiclesand the user may sync into the tank and/or weapon equipment to guide theuser in using the equipment while providing information and/or displaydata about the equipment. Such data may include the amount and efficacyof particular ammunition and the like. The eyepiece may displayinstructions to the soldier regarding safety procedures, missionspecifics, and the presences of enemies and friendlies in the area. Suchdisplay may include visual and audio information. In embodiments, theuser may stream a 360-degree view from the surrounding environment outside of the tank by using he eyepiece to sync into a camera or otherdevice with such a view. Video/audio feed may be provided to as manyusers inside of or outside of the tank/vehicle as necessary. This mayallow the user to monitor vehicle and stationary threats. The eyepiecemay communicate with the vehicle, and various vehicles, aircraft vesselsand devices as described herein or otherwise apparent to one of ordinaryskill in the art, to monitor vehicle statistics such as armor breach,engine status, and the like. The eyepiece may further provide GPS fornavigational purposes, and use of Black Silicon or other technology asdescribed herein to detect enemy and navigate to the environment atnight and in times of less than optimal viewing and the like.

Further, the eyepiece may be used in the tank/land vehicle environmentfor surveillance. In embodiments, the user may be able to sync intocameras or other devices to get a 360-degree field of view to gatherinformation. Night vision and/or SWIR and the like as described hereinmay be used for further information gathering where necessary. The usermay use the eyepiece to detect heat signatures to survey the environmentto detect potential threats, and may view soil density and the like todetect roadside bombs, vehicle tracks, various threats and the like.

In embodiments, the eyepiece may be used to facilitate group transportwith a tank or other land vehicle. For example the user may be providedwith a checklist that is visual, interactive or otherwise for items andpersonnel to be transported. The user may be able to track and update amanifest of items to track such as those in transport and the like. Theuser may be able to view maps of the surrounding area, scan papers anddocuments for identification of personnel, identify and track itemsassociated with individuals in transport, view the itinerary/missioninformation of the individual in transport and the like.

The eyepiece may also be used in maintenance of the vehicle. Forexample, it may provide the user with a pre-journey check for properfunctioning of the tank or other vehicle, it may alert the operation ofproper routine maintenance was performed or not completed prior to themission. Further, a user may be provided with a detailed history toreview maintenance performed and the like. In embodiments, the eyepiecemay also assist in maintaining the vehicle by providing an augmentedreality or other program that instructs the user in performing suchmaintenance.

In embodiments, the eyepiece may provide a user with various forms ofguidance when in an urban or suburban environment. For example, theeyepiece may be used in such environments when training, going intocombat, for surveillance, and the like. Such use may be suited forpersonnel of various ranks and levels.

By way of example, a user may use the eyepiece for training when incombat, under attack or a high stress situation, when interacting withlocal people, and the like in an urban or suburban environment. The usermay be presented with augmented reality situations, or otherwise, thatsimulate combat situations that may be seen when in such an environment.The training program may test the user on proper equipment and weaponuse and the like. The training program may be based on the user's ranksuch that his rank will determine the type of situation presented. Theuser's responses and actions may be recorded and or analyzed to providethe user with additional information, critique and to alter trainingexercises based on past data. In embodiments, the user may viewalternate scenarios of urban and suburban settings including actualbuildings and layouts of buildings and areas of potential combat. Theuser may be provided with climate and weather information prior to goinginto the area, and may be apprised of the number of people in the areaat a given time generally or at that time of day to prepare for possibleattacks or other engagement. Further, the user may be provided with thelocation of individuals in, around and atop of buildings in a given areaso that the user is prepared prior to entering the environment.

In urban and suburban environments, the eyepiece or other device mayallow the user to survey the local people as well. The user may be ableto gather face, iris, voice, and finger and palm print data of person'sof interest. The user may be able to scan such data without the user'sdetection from 0-5 meters, a greater distance or right next the POI. Inembodiments, the user may employ the eyepiece to see through smokeand/or destroyed environments, to note and record the presence ofvehicles in the area, to record environment images for future use suchas in battle plans, to note population density of an area at varioustimes of day, the lay out of various buildings and alleys, and the like.Furthermore, the user may gather and receive facts about a particularindigenous population with which the soldier will have contact.

The user may also employ the eyepiece or other device in urban/suburbanenvironments when in combat. The device may allow the user to use geolocation with a laser range finder to locate and kill an enemy target.In embodiments, it may give an areal view of the surrounding environmentand buildings. It may display enemy in the user's surrounding area andidentify the location of individuals such as enemies or friendlies orthose on the user's team. The user may use the eyepiece or other deviceto stay in contact with his home base, to view/hear instructions fromcommanding officers through the eyepiece where the instructions may bedeveloped after viewing or hearing data from the user's environment.Further, the eyepiece may also allow the user to give orders to otherson his team. In embodiments, the user may perform biometric datacollection on those in the vicinity, record such information and/orretrieve information about them for use in combat. The user may linkwith other soldier devices for monitoring and using various equipmentcarried by the soldier. In embodiments, the eyepiece may alert the userfor upcoming edges of buildings when on a roof top and alert whenapproaching a ground shift or ledge and the like. The use may be enabledto view a map overlay of the environment and the members of his team,and he may be able to detect nearby signals to be alerted and to alertothers of possible enemies in the vicinity. In various embodiments, theuser may use the eyepiece for communicating with other team members toexecute a plan. Further, the user may use the eyepiece to detect enemieslocated in dark tunnels and other areas where they may be located.

The eyepiece may also be used in a desert environment. In addition tothe general and/or applicable uses noted herein in relation to training,combat, survival, surveillance purposes, and the like, the eyepiece maybe further employed in various use scenarios that may be encountered inenvironments such as a desert environment. By way of example, when goinginto combat or training, the user may use the eyepiece to correctimpaired vision through sand storms in combat, surveillance, andtraining. Further, the eyepiece may simulate the poor visibility of sandstorms and other desert dangers for the user in training mode. Incombat, the eyepiece may assist the user in seeing or detecting theenemy in the presence of a sandstorm through various means as describedabove. Further, the user may be alerted to and/or be able to see thedifference between sand clouds caused by vehicles and those generated bythe wind in order to be alerted of potential enemy approach.

In various embodiments, the user may use the eyepiece to detect groundhazards and environmental hazards. For example the user may use theeyepiece to detect the edge of sand dunes, sand traps and the like. Theuser may also use the eyepiece to detect sand density to detect varioushazards such as ground holes, cliffs, buried devices such as landminesand bombs, and the like. The user may be presented with a map of thedesert to view the location of such hazards. In embodiments, the usermay be provided a means by which to monitor his vital signs and to givehim alerts when he is in danger to do the extreme environmentalconditions such as heat during the day, cold at night, fluctuatingtemperatures, dehydration and the like. Such alerts and monitoring maybe provided graphically in a user interface displayed in the eyepieceand/or via audio information.

In embodiments, the user may be presented with a map of the desert toview the location of his team, and he may use the eyepiece to detectnearby signals, or otherwise, to get alerts of possible enemy forcesthat may be displayed on the map or in an audio alert from an earpiece.In such embodiments, the user may have an advantage over his enemies ashe may have the ability to determine the location of his team andenemies in sandstorms, buildings, vehicles and the like. The user mayview a map of his location which may show areas in which the user hastraveled recently as one color and new areas as another. In this way orthrough other means, the device may allow the user to not get lost andor stay moving in the proper direction. In embodiments, the user may beprovided with a weather satellite overlay to warn the user of sandstorms and hazardous weather.

The eyepiece may also be used in a wilderness environment. In additionto the general and/or applicable uses noted herein in relation totraining, combat, survival, surveillance purposes, and the like, theeyepiece may be further employed in various use scenarios that may beencountered in environments such as a wilderness environment.

By way of example the user may use the eyepiece in training forpreparation of being in the wilderness. For example the user may employthe eyepiece to simulate varying degrees of wilderness environments. Inembodiments, the user may experience very thick and heavy trees/brushwith dangerous animals about and in other training environments, he maybe challenged with fewer places to hide from the enemy.

In combat, the user may use the eyepiece for various purposes. The usermay use the eyepiece to detect freshly broken twigs and branches todetect recent enemy presence. Further, the user may use the eyepiece todetect dangerous cliffs, caves, changes in terrain, recentlymoved/disturbed dirt and the like. By way of example, by detecting thepresence of recently disturbed dirt, which may be detected if it has adifferent density or heat signature from the surrounding dirt/leaves orwhich may be detected by other means, the user may be alerted to a trap,bomb or other dangerous device. In various environments describedherein, the user may use the eyepiece to communicate with his team via auser interface or other means such that communication may remain silentand/or undetected by the enemy in close environments, open environmentssusceptible to echo, and the like. Also, in various environments, theuser may employ night vision as described herein to detect the presenceof enemies. The user may also view an overlay of trail maps and/ormountain trail maps in the eyepiece so that the user may view a pathprior to encountering potentially dangerous terrain and or situationswhere the enemy may be located. In various environments as describedherein, the eyepiece may also amplify the user's hearing for thedetection of potential enemies.

In embodiments, a user may employ the eyepiece in a wildernessenvironment in a search and rescue use scenario. For example, the usermay use the eyepiece to detect soil/leaf movement to determine if it'sbeen disturbed for tracking human tracks and for finding a buried body.The user may view a map of the area which has been tagged to show areasalready covered by air and or other team member searches to direct theuser from areas already scoured and toward areas not searched. Further,the user may use the eyepiece for night vision for human and/or animaldetection through trees, brush, thickets and the like. Further, by usingthe eyepiece to detect the presence of freshly broken twigs, the usermay be able to detect the presence or recent presence of persons ofinterest when in a surveillance and/or rescue mission. In embodiments,the user may also view an overlay of trail maps and/or mountain trailmaps in the eyepiece s so that the user may view a path prior toencountering potentially dangerous terrain and or situations.

In yet other embodiments, a user may employ the use of the eyepiece in awilderness for living off of the land and survival-type situations. Byway of example, the user may use the eyepiece to track animal presenceand movement when hunting for food. Further, the user may use theeyepiece for detection of soil moisture and to detect the presence andlocation of a water supply. In embodiments, the eyepiece may alsoamplify the user's hearing to detect potential prey.

The eyepiece may also be used in an artic environment. In addition tothe general and/or applicable uses noted herein in relation to training,combat, survival, surveillance purposes, and the like, the eyepiece maybe further employed in various use scenarios that may be encountered inenvironments such as an arctic environment. For example, when intraining, the eyepiece may simulate visual and audio white outconditions that a user may encounter in an arctic environment so thatthe user may adapt to operating under such stresses. Further, theeyepiece may provide the user with a program that simulates variousconditions and scenarios due to extreme cold that he may encounter, andthe program may track and display data related to the user's predictedloss of heat. Further, the program may adapt to simulate such conditionsthat the user would experience with such heat loss. In embodiments, theprogram may simulate the inability of the user to control his limbsproperly which may manifest in a loss of weapon accuracy. In otherembodiments, the user may be provided life saving information andinstructions about such things as burrowing in the snow for warmth, andvarious survival tips for artic conditions. In yet other embodiments,the eyepiece may sync into a vehicle such that the vehicle responds asif the vehicle were performing in a particular environment, for examplewith artic conditions and snow and ice. Accordingly the vehicle mayrespond to the user as such and the eyepiece may also simulate visualand audio as if the user were in such an environment.

In embodiments, the user may use the eyepiece in combat. The soldier mayuse the eyepiece to allow him to see through white out conditions. Theuse may be able to pull up an overlay map and/or audio that provides ainformation of buildings ditches, land hazards and the like to allow thesoldier to move around the environment safely. The eyepiece may alertthe user to detections in the increase or decrease of snow density tolet him know when the landmass under the snow has changed such as todenote a possible ditch, hole or other hazard, an object buried in thesnow and the like. Further, in conditions where it is difficult to see,the user may be provided with the location of his team members andenemies whether or not snow has obstructed his view. The eyepiece mayalso provide heat signatures to display animals and individuals to theuser in an artic environment. In embodiments, a user interface in theeyepiece may show a soldier's his vitals and give alerts when he is indanger doe to the surrounding extreme environmental conditions.Furthermore, the eyepiece may help the user operate a vehicle in snowyconditions by providing alerts from the vehicle to the user regardingtransmission slipping, wheel spinning, and the like.

The eyepiece may also be used in a jungle environment. In addition tothe general and/or applicable uses noted herein in relation to training,combat, survival, surveillance purposes, and the like, the eyepiece maybe further employed in various use scenarios that may be encountered inenvironments such as a jungle environment. For example the eyepiece maybe employed in training to provide the user with information regardingwhich plants may be eaten, which are poisonous and what insects andanimals may present the user with danger. In embodiments, the eyepiecemay simulate various noises and environments the user may encounter inthe jungle so that when in battle the environment is not a distraction.Further, when in combat or an actual jungle environment, the user may beprovided with a graphical overlay or other map to show him thesurrounding area and/or to help him track where he's been and where hemust go. It may alert him of allies and enemies in the area, and it maysense movement in order to alert the user of potential animals and/orinsects nearby. Such alerts may help the user survive by avoiding attackand finding food. In other embodiments, the user may be provided withaugmented reality data such as in the form of a graphical overlay thatallows the user to compare a creature and/or animal to those encounteredto help the user discern which are safe for eating, which are poisonousand the like. By having information that a particular creature is not athreat to the user, he may be spared of having to deploy a weapon whenin stealth or quiet mode.

The eyepiece may also be used in relation to Special Forces missions. Inaddition to the general and/or applicable uses noted herein in relationto training, combat, survival, surveillance purposes, and the like, theeyepiece may be further employed in various use scenarios that may beencountered in relation to special forces missions. In embodiments, theeyepiece may be of particular use on stealth missions. For example, theuser may communicate with his team in complete silence through a userinterface that each member may see on his eyepiece. The user sharinginformation may navigate through the user interface with eye movementsand/or a controller device and the like. As the user puts upinstructions and/or navigates through the user interface and particulardata concerning the information to convey, the other users may see thedata as well. In embodiments, various users may be able to insertquestions via the user interface to be answered by the instructionleader. In embodiments, a user may speak or launch other audio that allusers may hear through their eyepiece or other device. This may allowusers in various locations on the battlefield to communicate battleplans, instructions, questions, share information and the like and mayallow them to do so without being detected.

In embodiments, the eyepiece may also be used for military firefighting. By way of example, the user may employ the eyepiece to run asimulation of firefighting scenarios. The device may employ augmentedreality to simulate fire and structural damage to a building as timegoes by and it may otherwise recreate life-like scenarios. As notedherein, the training program may monitor the user's progress and/oralter scenarios and training modules based on the user's actions. Inembodiments, the eyepiece may be used in actual firefighting. Theeyepiece may allow the user to see though smoke through various means asdescribed herein. The user may view, download or otherwise, access alayout of the building, vessel, aircraft vehicle or structure that's onfire. In embodiments, the user will have an overview map or other mapthat displays where each team member is located. The eyepiece maymonitor the user-worn or other devices during firefighting. The user maysee his oxygen supply levels in his eyepiece and may be alerted as towhen he should come out for more. The eyepiece may send notificationsfrom the user's devices to the command outside of the structure todeploy new personnel to come in or out of the fire and to give statusupdates and alert of possible fire fighter danger. The user may have hisvital signs displayed to determine if he is overheating, losing too muchoxygen and the like. In embodiments, the eyepiece may be used to analyzewhether cracks in beams or forming based on beam density, heatsignatures and the like and inform the user of the structural integrityof the building or other environment. The eyepiece may provide automaticalerts when structural integrity is compromised.

In embodiments, the eyepiece may also be used for maintenance purposes.For example, the eyepiece may provide the user with a pre-mission and/oruse checklist for proper functioning of the item to be used. It mayalert the operator if proper maintenance has not been logged in theitem's database. It may provide a virtual maintenance and/or performancehistory for the user to determine the safety of the item or of necessarymeasures to be taken for safety and/or performance. In embodiments, theeyepiece may be used to perform augmented reality programs and the likefor training the user in weapon care and maintenance and for lessons inthe mechanics of new and/or advanced equipment. In embodiments, theeyepiece may be used in maintenance and/or repair of various items suchas weapons, vehicles, aircraft, devices and the like. The user may usethe eyepiece to view an overlay of visual and/or audio instructions ofthe item to walk the user through maintenance without the need for ahandheld manual. In embodiments, video, still images, 3D and/or 2Dimages, animated images, audio and the like may be used for suchmaintenance. In embodiments, the user may view an overlay and/or videoof various images of the item such that the user is shown what parts toremove, in what order, and how, which parts to add, replace, repair,enhance and the like. In embodiments such maintenance programs may beaugmented reality programs or otherwise. In embodiments, the user mayuse the eyepiece to connect with the machine or device to monitor thefunctioning and or vital statistics of the machine or device to assistin repair and/or to provide maintenance information. In embodiments, theuser may be able to use the eyepiece to propose a next course of actionduring maintenance and the eyepiece may send the user information on thelikelihood of such action harming the machine, helping to fix themachine, how and/or if the machine will function after the next step andthe like. In embodiments, the eyepiece may be used for maintenance ofall items, machines, vehicles, devices, aircraft and the like asmentioned herein or otherwise applicable to or encountered in a militaryenvironment.

The eyepiece may also be used in environments where the user has somedegree of unfamiliarity with the language spoken. By way of example, asoldier may use the eyepiece and/or device to access near real-timetranslation of those speaking around him. Through the device's earpiece,he may hear a translation in his native language of one speaking to him.Further, he may record and translate comments made by prisoners and/orother detainees. In embodiments, the soldier may have a user interfacethat enables translating a phrase or providing translation to the uservia an earpiece, via the user's eyepiece in a textual image orotherwise. In embodiments, the eyepiece may be used by a linguist toprovide a skilled linguist with supplemental information regardingdialect spoken in a particular area or that which is being spoken bypeople near him. In embodiments, the linguist may use the eyepiece torecord language samples for further comparison and/or study. Otherexperts may use the eyepiece to employ voice analysis to determine ifthe speaker is experiencing anger, shame, lying, and the like bymonitoring inflection, tone, stutters and the like. This may give thelistener native the speaker's intentions even when the listener andspeaker speak different languages.

In embodiments, the eyepiece may allow the user to decipher bodylanguage and/or facial expressions or other biometric data from another.For example, the user may use the device to analyze a person's pupildilation, eye blink rates, voice inflection, body movement and the liketo determine if the person is lying, hostile, under stress, likely athreat, and the like. In embodiments, the eyepiece may also gather datasuch as that of facial expressions to detect and warn the user if thespeaker is lying or likely making unreliable statements, hostile, andthe like. In embodiments, the eyepiece may provide alerts to the userwhen interacting with a population or other individuals to warn aboutpotential threatening individuals that may be disguised as non-combativeor ordinary citizens or other individuals. User alerts may be audioand/or visual and may appear in the user's eyepiece in a user interfaceor overlaid in the user's vision and/or be associated with the surveyedindividual in the user's line of vision. Such monitoring as describedherein may be undetected as the user employs the eyepiece and/or deviceto gather the data from a distance or it may be performed up-close in adisguised or discrete fashion, or performed with the knowledge and/orconsent of the individual in question.

The eyepiece may also be used when dealing with bombs and otherhazardous environments. By way of example, the eyepiece may provide auser with alerts of soil density changes near the roadside which couldalert the user and/or team of a buried bomb. In embodiments, similarlymethods may be employed in various environments, such as testing thedensity of snow to determine if a bomb or other explosive may be foundin artic environments and the like. In embodiments, the eyepiece mayprovide a density calculation to determine whether luggage and/ortransport items tend to have an unexpected density or one that fallsoutside of a particular range for the items being transported. Inembodiments, the eyepiece may provide a similar density calculation andprovide an alert if the density is found to be one that falls withinthat expected for explosive devices, other weapons and the like. Oneskilled in the art will recognize that bomb detection may be employedvia chemical sensors as well and/or means known in the art and may beemployed by the eyepiece in various embodiments. In embodiments, theeyepiece may be useful in bomb disposal. The user may be provided withan augmented reality or other audio and/or visual overlay in order togain instructions on how to diffuse the particular type of bomb present.Similar to the maintenance programs described above, the user may beprovided with instructions for diffusing a bomb. In embodiments, if thebomb type is unknown a user interface may provide the user withinstructions for safe handling and possible next steps to be taken. Inembodiments, the user may be alerted of a potential bomb in the vicinityand may be presented with instructions for safe dealing with thesituation such as how to safely flee the bomb area, how to safely exit avehicle with a bomb, how closely the user may come to the bomb safely,how to diffuse the bomb via instructions appropriate for the situationand the user's skill level, and the like. In embodiments, the eyepiecemay also provide a user with training in such hazardous environments andthe like.

In embodiments, the eyepiece may detect various other hazards such asbiological spills, chemical spills, and the like and provide the userwith alerts of the hazardous situation. In embodiments, the user mayalso be provided with various instructions on diffusing the situation,getting to safety and keeping others safe in the environment and/orunder such conditions. Although situations with bombs have beendescribed, it is intended that the eyepiece may be used similarly invarious hazardous and/or dangerous situations and to guard against andto neutralize and/or provide instruction and the like when such dangerand hazards are encountered.

The eyepiece may be used in a general fitness and training environmentin various embodiments. The eyepiece may provide the user with suchinformation as the miles traveled during his run, hike, walk and thelike. The eyepiece may provide the user with information such as thenumber of exercised performed, the calories burned, and the like. Inembodiments, the eyepiece may provide virtual instructions to the userin relation to performing particular exercises correctly, and it mayprovide the user with additional exercises as needed or desired.Further, the eyepiece may provide a user interface or otherwise wherephysical benchmarks are disclosed for the soldier to meet therequirements for his particular program. Further, the eyepiece mayprovide data related to the amount and type of exercise needed to becarried out in order for user to meet such requirements. Suchrequirements may be geared toward Special Forces qualification, basictraining, and the like. In embodiments, the user may work with virtualobstacles during the workout to prevent the user from setting up actualhurdles, obstacles and the like.

Although specific various environments and use scenarios have beendescribed herein, such description is not intended to be limiting.Further, it is intended that the eyepiece may be used in variousinstances apparent to one of ordinary skill in the art. It is alsointended that applicable uses of the eyepiece as noted for particularenvironments may be applied in various other environments even thoughnot specifically mentioned therewith.

In embodiments, a user may access and/or otherwise manipulate a libraryof information stored on a secure digital (SD) card, Mini SD card, othermemory, remotely loaded over a tactical network, or stored by othermeans. The library may be part of the user's equipment and/or it may beremotely accessible. The user's equipment may include a DVR or othermeans for storing information gathered by the user and the recorded dataand/or feed may be transmitted elsewhere as desired. In embodiments, thelibrary may include images of local threats, information and/or imagesof various persons listed as threats and the like. The library ofthreats may be stored in an onboard mini-SD card or other means. Inembodiments, it may be remotely loaded over a tactical network.Furthermore, in embodiments, the library of information may containprograms and other information useful in the maintenance of militaryvehicles or the data may be of any variety or concerning any type ofinformation. In various embodiments, the library of information may beused with a device such that data is transferred and/or sent to or fromthe storage medium and the user's device. By way of example, data may besent to a user's eyepiece and from a stored library such that he is ableto view images of local persons of interest. In embodiments, data may besent to and from a library included in the soldier's equipment orlocated remotely and data may be sent to and from various devices asdescribed here. Further, data may be sent between various devices asdescribed herein and various libraries as described above.

In embodiments, military simulation and training may be employed. By wayof example, gaming scenarios normally used for entertainment may beadapted and used for battlefield simulation and training. Variousdevices, such as the eyepiece described herein may be used for suchpurpose. Near field communications may be used in such simulation toalert personnel, present dangers, change strategy and scenario and forvarious other communication. Such information may be posted to shareinformation where it is needed to give instruction and/or information.Various scenarios, training modules and the like may be run on theuser's equipment. For example only, and not to limit the use of suchtraining, a user's eyepiece may display an augmented reality battleenvironment. In embodiments, the user may act and react in such anenvironment as if he were actually in battle. The user may advance orregress depending on his performance. In various embodiments, the user'sactions may be recorded for feedback to be provided based on hisperformance. In embodiments, the use may be provided with feedbackindependent of whether his performance was recorded. In embodiments,information posted as described above may be password or biometricallyprotected and or encrypted and instantly available or available after aparticular period of time. Such information stored in electronic formmay be updated instantly for all the change orders and updates that maybe desired.

Near field communications or other means may also be used in trainingenvironments and for maintenance to share and post information where itis needed to give instruction and/or information. By way of example,information may be posed in classrooms, laboratories maintenancefacilitates, repair bays, and the like or wherever it is needed for suchtraining and instruction. A user's device, such as the eyepiecedescribed herein, may allow such transmission and receipt ofinformation. Information may be shared via augmented reality where auser encounters a particular area and once there he is notified of suchinformation. Similarly as descried herein, near field communications maybe used in maintenance. By way of example, information may be postedprecisely where it is needed, such as in maintenance facilities, repairbays, associated with the item to be repaired, and the like. Morespecifically, and not to limit the present disclosure, repairinstructions may be posted under the hood of a military vehicle andvisible with the use of the soldier's eyepiece. Similarly, variousinstruction and training information may be shared with various users inany given training situation such as training for combat and/or trainingfor military device maintenance. In embodiments, information posted asdescribed above may be password or biometrics protected and or encryptedand instantly available or available after a particular period of time.Such information stored in electronic form may be updated instantly forall the change orders and updates that may be desired.

In embodiments, an application applied to the present invention may befor facial recognition or sparse facial recognition. Such sparse facialrecognition may use one or more facial features to exclude possibilitiesin identifying persons of interest. Space facial recognition may haveautomatic obstruction masking and error and angle correction. Inembodiments, and by way of example and not to limit the presentinvention, the eyepiece, flashlight and devices as described herein mayallow for sparse facial recognition. This may work like human vision andquickly exclude regions or entire profiles that don't match by usingsparse matching on all image vectors at once. This may make it almostimpossible for false positives. Further, this may simultaneously utilizemultiple images to enlarge the vector space and increase accuracy. Thismay work with either multiple database or multiple target images basedon availability or operational requirement. In embodiments, a device maymanually or automatically identify one or more specific clean featureswith minimal reduction in accuracy. By way of example, accuracy may beof various ranges and it may be at least 87.3% for a nose, 93.7% for aneye, and 98.3% for a mouth and chin. Further angle correction withfacial reconstruction may be employed and, in embodiments, up to a 45degree off angle correction with facial reconstruction may be achieved.This may be further enhanced with 3D image mapping technology. Further,obscured area masking and replacement may be employed. In embodiments,97.5% and 93.5% obscured area masking and replacement may be achievedfor sunglasses and a scarf respectively. In embodiments, the ideal inputimage may be 640 by 480. The target image may match reliably with lessthan 10% of the input resolution due to long range or atmosphericobscurants. Further, the specific ranges as noted above may be greateror lesser in various embodiments.

In various embodiments, the devices and/or networks described herein maybe applied for the identification and or tracking of friends and/orallies. In embodiments, facial recognition may be employed to positivelyidentify friends and or friendly forces. Further, real-time networktracking and/or real-time network tracking of blue and red forces mayallow a user to know where his allies and/or friendlies are. Inembodiments, there may be a visual separation range between blue and redforces and/or forces identified by various markers and/or means.Further, the user may be able to geo-locate the enemy and share theenemy's location in real-time. Further, the location of friendlies maybe shared in real time as well. Devices used for such an application maybe biometric collection glasses, eyepiece other devices as describedherein and those known to one of ordinary skill in the art.

In embodiments, the devices and/or networks described herein may beapplied in medical treatment in diagnosis. By way of example, suchdevices may enable medical personnel to make remote diagnoses. Further,and by way of example, when field medics arrive on a scene, or remotely,they may use a device such as a fingerprint sensor to instantaneouslycall up the soldier's medical history, allergies, blood type and othertime sensitive medical data to apply the most effective treatment. Inembodiment, such data may be called up via facial recognition, irisrecognition, and the like of the soldier which may be accomplished viathe eyepiece described herein or another device.

In embodiments, users may share various data via various networks anddevices as described herein. By way of example, a 256-bit AES encryptedvideo wireless transceiver may bi-directionally share video betweenunits and/or with a vehicle's computer. Further, biometric collection ofdata, enrollment, identification and verification of potential personsof interest, biometric data of persons of interest and the like may beshared locally and/or remotely over a wireless network. Further, suchidentification and verification of potential persons of interest may beaccomplished or aided by the data shared locally and/or remotely over awireless network. The line of biometric systems and devices as describedherein may be enabled to share data over a network as well. Inembodiments, data may be shared with, from and/or between variousdevices, individuals, vehicles, locations, units and the like. Inembodiments there may be inter-unit and intra unit communication anddata sharing. Data may be shared via, from and/or between existingcommunications assets, a mesh network or other network, a mil-con typeultra wide band transceiver caps with 256-bit encryption, a mil-con typecable, removable SD and/or microSD memory card, a Humvee, PSD S2,unmanned aerial vehicle, WBOTM, or other network relay, a combat radio,a mesh networked computer, devices such as but not limited to variousdevices described herein, a bio-phone 3G/4G networked computer, adigital dossier, tactical operating centers, command posts, DCSG-A, BATservers, individuals and/or groups of individuals, and any eyepieceand/or device described herein and/or those known to persons skilled inthe art and the like.

In embodiments, a device as described herein or other device may containa viewing pane that reverses to project imagery on any surface forcombat team viewing by a squad and/or team leader. The transparentviewing pane or other viewing pane may be rotated 180 degrees or anotherquantity of degrees in projection mode to share data with a team and/orvarious individuals. In embodiments, devices including but not limitedto a monocular and binocular NVG may interface with all or virtually alltactical radios in use and allow the user to share live video, S/A,biometric data and other data in real-time or otherwise. Such devices asthe binocular and monocular noted above may be a, VIS, NIR and/or SWIRbinocular or monocular that may be self-contained, and comprise a colorday/night vision and/or digital display with a compact, encrypted,wireless-enabled computer for interfacing with tactical radios. Variousdata may be shared over combat radios, mesh networks and long-rangetactical networks in real time or near real time. Further, data may beorganized into a digital dossier. Data of a person of interest (POI) maybe organized into a digital dossier whether such POI rest was enrolledor not. Data that is shared, in embodiments, may be compared,manipulated and the like. While specific devices are mentioned, anydevice mentioned herein may be capable of sharing information asdescribed herein and/or as would be recognized by one having ordinaryskill in the art.

In embodiments, biometric data, video, and various other types of datamay be collected via various devices, methods and means. For example,fingerprints and other data may be collected from weapons and otherobjects at a battle, terrorism and/or crime scene. Such collection maybe captured by video or other means. A pocket bio cam, flashlight asdescribed herein with built in still video camera, various other devicesdescribed herein, or other device may collect video, record, monitor,and collect and identify biometric photographic data. In embodiments,various devices may record, collect, identify and verify data andbiometric data relating to the face, fingerprints, latent fingerprints,latent palm prints, iris, voice, pocket litter, scars, tattoos, andother identifying visible marks and environmental data. Data may begeo-located and date/time stamped. The device may capture EFTS/EBTScompliant salient images to be matched and filed by any biometricmatching software. Further, video scanning and potential matchingagainst a built-in or remote iris and facial database may be performed.In embodiments, various biometric data may be captured and/or comparedagainst a database and/or it may be organized into a digital dossier. Inembodiments, an imaging and detection system may provide for biometricsscanning and may allow facial tracking and iris recognition of multiplesubjects. The subjects may be moving in or out of crowds at high speedsand may be identified immediately and local and/or remote storage and/oranalysis may be performed on such images and/or data. In embodiments,devices may perform multi-modal biometric recognition. For example, adevice may collect and identify a face and iris, an iris and latentfingerprints, various other combinations of biometric data, and thelike. Further, a device may record video, voice, gait, fingerprints,latent fingerprints, palm prints, latent palm prints and the like andother distinguishing marks and/or movements. In various embodiments,biometric data may be filed using the most salient image plus manualentry, enabling partial data capture. Data may be automaticallygeo-located, time/date stamped and filed into a digital dossier with alocally or network assigned GUID. In embodiments, devices may recordfull livescan 4 fingerprint slaps and rolls, fingerprint slaps androlls, palm prints, finger tips and finger prints. In embodiments,operators may collect and verify POIs with an onboard or remote databasewhile overseeing indigenous forces. In embodiments, a device may accessweb portals and biometric enabled watch list databases and/or maycontain existing biometric pre-qualification software for POIacquisition. In embodiments, biometrics may be matched and filed by anyapproved biometric matching software for sending and receiving secureperishable voice, video and data. A device may integrate and/orotherwise analyze biometric content. In embodiments, biometric data maybe collected in biometric standard image and data formats that can becross referenced for a near real or real time data communication withthe Department of Defense Biometric Authoritative or other data base. Inembodiments, a device may employ algorithms for detection, analysis, orotherwise in relation to finger and palm prints, iris and face images. Adevice, in embodiments, may illuminate an iris or latent fingerprintsimultaneously for a comprehensive solution. In embodiments, a devicemay use high-speed video to capture salient images in unstablesituations and may facilitate rapid dissemination of situationalawareness with intuitive tactical display. Real time situationalawareness may be provided to command posts and/or tactical operatingcenters. In embodiments, a device may allow every soldier to be a sensorand to observe and report. Collected data may be tagged with date, timeand geo-location of collection. Further, biometric images may beNIST/ISO compliant, including ITL 1-2007. Further, in embodiments, alaser range finder may assist in biometric capture and targeting. Alibrary of threats may be stored in onboard Mini-SD card or remotelyloaded over a tactical network. In embodiments, devices may wirelesslytransfer encrypted data between devices with a band transceiver and/orultrawide band transceiver. A device may perform onboard matching ofpotential POI's against a built in database or securely over abattlefield network. Further, a device may employ high-speed video tocapture salient images in all environmental conditions. Biometricprofiles may be uploaded downloaded and searched in seconds or less. Inembodiments, a user may employ a device to geo-locate a POI with visualbiometrics at a safe distance and positively identify a POI with robustsparse recognition algorithms for the face, iris and the like. Inembodiments, a user may merge and print a visual biometrics on onecomprehensive display with augmented target highlighting and viewmatches and warnings without alerting the POI. Such display may be invarious devices such as an eyepiece, handheld device and the like.

In embodiments, as indigenous persons filter through a controlledcheckpoint and/or vehicle stops, an operator can collect, enroll,identify and verify POIs from a watch list using low profile face andiris biometrics. In embodiments, biometric collection and identificationmay take place at a crime scene. For example an operator may rapidlycollect biometric data from all potential POIs at a bombing or othercrime scene. The data may be collected, geo-tagged and stored in adigital dossier to compare POIs against past and future crime scenes.Further, biometric data may be collected in real time from POIs in houseand building searches. Such data displayed may let the operator knowwhether to release detain or arrest a potential POI. In otherembodiments, low profile collection of data and identification may occurin street environments or otherwise. A user may move through a marketplace for example and assimilate with the local population whilecollecting biometric, geo-location and/or environmental data withminimal visible impact. Furthermore, biometric data may be collected onthe dead or wounded to identify whether they were or are a POI. Inembodiments, a user may identify known or unknown POI's by facialidentification, iris identification, fingerprint identification, visibleidentifying marks, and the like of the deceased or wounded, or othersand keep a digital dossier updated with such data.

In embodiments, a laser range finder and/or inclinometer may be used todetermine the location of persons of interest and/or improvisedexplosive devices, other items of interest, and the like. Variousdevices described herein may contain a digital compass, inclinometer anda laser range finder to provide geo-location of POIs, targets, IEDs,items of interest and the like. The geo-location of a POI and/or item ofinterest may be transmitted over networks, tactical networks, orotherwise, and such data may be shared among individuals. Inembodiments, a device may allow an optical array and a laser rangefinder to geo-locate and range multiple POIs simultaneously withcontinuous observation of a group or crowd in the field in anuncontrolled environment. Further, in embodiments, a device may containa laser range finder and designator to range and paint a targetsimultaneously with continuous observation of one or more targets.Further, in embodiments, a device may be soldier-worn, handheld orotherwise and include target geo-location with integrated laser rangefinder, digital compass, inclinometer and GPS receiver to locate theenemy in the filed. In embodiments, a device may contain an integrateddigital compass, inclinometer, MEMs Gyro and GPS receiver to record anddisplay the soldier's position and direction of his sight. Further,various devices may include an integrated GPS receiver or other GPSreceiver, IMU, 3-axis digital compass or other compass, laser rangefiner, gyroscope, micro-electro-mechanical system based gyroscope,accelerometer and/or an inclinometer for positional and directionalaccuracy and the like. Various devices and methods as described hereinmay enable a user to locate enemy and POIs in the filed and share suchinformation with friendlies via a network or other means.

In embodiments, users may be mesh networked or networked together withcommunications and geo-location. Further, each user may be provided witha pop-up, or other location map of all users or proximate users. Thismay provide the user with knowledge of where friendly forces arelocated. A described above, the location of enemies may be discovered.The location of enemies may be tracked and provided with a pop-up orother location map of enemies which may provide the user with knowledgeof where friendly forces are located. Location of friendlies and enemiesmay be shared in real time. Users may be provided with a map depictingsuch locations. Such maps of the location and/or number of friendlies,enemies and combinations thereof may be displayed in the user's eyepieceor other device for viewing.

In embodiments, devices, methods, and applications may allow forhands-free, wireless, maintenance and repair visually and/or audioenhanced instructions. Such applications may include RFID sensing forparts location and kitting. In examples, a user may use a device foraugmented reality guided filed repair. Such filed repair may be guidedby hands-free, wireless, maintenance and repair instructions. A device,such as an eyepiece, projector, monocular and the like and/or otherdevices as described herein may display images of maintenance and repairprocedures. In embodiments, such images may be still and/or video,animated, 3-D, 2-D, and the like. Further, the user may be provided withvoice and/or audio annotation of such procedures. In embodiments, thisapplication may be used in high threat environments where workingundetected is a safety consideration. Augmented reality images and videomay be projected on or otherwise overlaid on the actual object withwhich the user is working or in the user's field of view of the objectto provide video, graphical, textual or other instructions of theprocedure to be performed. In embodiments, a library of programs forvarious procedures may be downloaded and accessed wired or wirelesslyfrom a body worn computer or from a remote device, database and/orserver, and the like. Such programs may be used for actual maintenanceor training purposes.

In embodiments, the devises, methods and descriptions found herein mayprovide for an inventory tracking system. In embodiments, such trackingsystem may allow a scan from up to 100 m distance to handle more than1000 simultaneous links with 2 mb/s data rate. The system may giveannotated audio and/or visual information regarding inventory trackingwhen viewing and/or in the vicinity of the inventory. In embodiments,devices may include an eyepiece, monocular, binocular and/or otherdevices as described herein and inventory tracking may use SWIR, SWIRcolor, and/or night vision technology, body worn wired or wirelesscomputers, wireless UWB secure tags, RFID tags, a helmet/hardhat readerand display and the like. In embodiments, and by way of example only, auser may receive visual and/or audio information regarding inventorysuch as which items are to be destroyed, transferred, the quantity ofitems to be destroyed or transferred, where the items are to betransferred or disposed and the like. Further, such information mayhighlight, or otherwise provide a visual identification of the items inquestion along with instructions. Such information may be displayed on auser's eyepiece, projected onto an item, displayed on a digital or otherdisplay or monitor and the like. The items in question may be tagged viaUWB and/or RFID tags, and/or augmented reality programs may be used toprovide visualization and/or instruction to the user such that thevarious devices as described herein may provide the information asnecessary for inventory tracking and management.

In various embodiments, SWIR, SWIR color, monocular, night vision, bodyworn wireless computer, the eyepiece as described herein and/or devicesas described herein may be used when firefighting. In embodiments, auser may have increased visibility through smoke, and the location ofvarious individuals may be displayed to the user by his device in anoverlaid map or other map so that he may know the location offirefighters and/or others. The device may show real-time display of allfirefighters' locations and provide hot spot detection of areas withtemperatures of less than and greater than 200 degrees Celsius withouttriggering false alarms. Maps of the facility may also be provided bythe device, displayed on the device, projected from the device and/oroverlaid in the user's line of site through augmented reality or othermeans to help guide the user through the structure and/or environment.

Systems and devices as described herein may be configurable to anysoftware and/or algorithm to conform to mission specific needs and/orsystem upgrades.

Referring to FIG. 73, the eyepiece 100 may interface with a ‘biometricflashlight’ 7300, such as including biometric data taking sensors forrecording an individual's biometric signature(s) as well as the functionand in the form factor of a typical handheld flashlight. The biometricflashlight may interface with the eyepiece directly, such as though awireless connection directly from the biometric flashlight to theeyepiece 100, or as shown in the embodiment represented in FIG. 73,through an intermediate transceiver 7302 that interfaces wirelessly withthe biometric flashlight, and through a wired or wireless interface fromthe transceiver to the eyepiece (e.g. where the transceiver device isworn, such as on the belt). Although other mobile biometric devices aredepicted in figures without showing the transceiver, one skilled in theart will appreciate that any of the mobile biometric devices may be madeto communicate with the eyepiece 100 indirectly through the transceiver7300, directly to the eyepiece 100, or operate independently. Data maybe transferred from the biometric flashlight to the eyepiece memory, tomemory in the transceiver device, in removable storage cards 7304 aspart of the biometric flashlight, and the like. The biometric flashlightmay include an integrated camera and display, as described herein. Inembodiments, the biometric flashlight may be used as a stand-alonedevice, without the eyepiece, where data is stored internally andinformation provided on a display. In this way, non-military personnelmay more easily and securely use the biometric flashlight. The biometricflashlight may have a range for capturing curtain types of biometricdata, such as a range of 1 meter, 3 meters, 10 meters, and the like. Thecamera may provide for monochrome or color images. In embodiments, thebiometric flashlight may provide a covert biometric data collectionflashlight-camera that may rapidly geo-locate, monitor and collectenvironmental and biometric data, for onboard or remote biometricmatching. In an example use scenario, a soldier may be assigned to aguard post at nighttime. The soldier may utilize the biometricflashlight seemingly only as a typical flashlight, but where unbeknownstto the individuals being illuminated by the device, is also runningand/or taking biometrics as part of a data collection and/or biometricsidentification process.

Referring now to FIG. 76, a 360° imager utilizes digital foveatedimaging to concentrates pixels to any given region, delivering a highresolution image of the specified region. Embodiments of the 360° imagermay feature continuous 360°×40° panoramic FOV with super-high resolutionfoveated view and simultaneous and independent 10× optical zoom. The360° imager may include dual 5 megapixel sensors and imagingcapabilities of 30 fps and image acquisition time <100. The 360° imagermay include a gyro-stabilized platform with independently stabilizedimage sensors. The 360° imager may have only one moving part and twoimaging sensors that allows for reduced image processing bandwidth in acompact optical system design. The 360° image may also feature lowangular resolution and high-speed video processing and may be sensoragnostic. The 360° image may be used as a surveillance fixture in afacility, on a mobile vehicle with a gyro stabilized platform, mountedon a traffic light or telephone pole, robot, aircraft, or other locationthat allows for persistent surveillance. Multiple users mayindependently and simultaneously view the environment imaged by the 360°imager. For example, imagery captured by the 360° imager may bedisplayed in the eyepiece to allow all recipients of the data, such asall occupants in a combat vehicle, to have real-time 360° situationalawareness. The panoramic 360° imager may recognize a person at 100meters and foveated 10× zoom can be used to read a license plate at 500meters. The 360° imager allows constant recording of the environment andfeatures an independent controllable foveated imager.

FIG. 76A depicts an assembled 360° imager and FIG. 76B depicts a cutawayview of the 360° imager. The 360° imager include a capturing mirror7602, objective lens 7604, beam splitter 7608, lenses 7610 and 7612,MEMS mirror 7614, panoramic sensor 7618, panoramic image lens 7620,folding mirror 7622, foveation sensor 7624, and foveated image lens7628. Imagery collected with the 360° imager may be geo-located and timeand date stamped. Other sensors may be included in the 360° imager, suchas thermal imaging sensor, NIR sensor, SWIR sensor, and the like. TheMEMS mirror 7614 is a unique mirror prism that uses a single-viewpointhemispherical capture system allowing for high and uniform resolution.The imager design enables <0.1° scanning accuracy, foveated distortion<1%, 50% MTF @400 lp/mm, and foveated acquisition <30 milliseconds.

The 360° imager may be part of a network with wireless or physical reachback to a TOC or database. For example, a user may use a display with a360° imager driver to view imagery from a 360° imager wirelessly orusing a wired connection, such as a mil-con type cable. The display maybe a combat radio or mesh networked computer that is networked with aheadquarters. Data from a database, such as a DoD authoritative databasemay be accessed by the combat radio or mesh networked computer, such asby using a removable memory storage card or through a networkedconnection.

Referring now to FIG. 77, a multi-coincident view camera may be used forimaging. The feed from the multi-coincident view camera may betransmitted to the eyepiece 100 or any other suitable display device. Inone embodiment, the multi-coincident view camera may be afully-articulating, 3- or 4-coincident view, SWIR/LWIR imaging, andtarget designating system that allows simultaneous: wide, medium andnarrow field-of-view surveillance, with each sensor at VGA or SXVGAresolution for day or night operations. The lightweight, gimbaled sensorarray may be inertially stabilized as well as geo-referenced enabling ahighly accurate sensor positioning and target designating with its NVGcompatible laser pointer capability in all conditions. Its uniquemultiple and simultaneous fields-of-view enable wide area surveillancein the visible, near-infrared, short wave infrared and long waveinfrared regions. It also permits a high resolution, narrowfield-of-view for more precise target identification and designationwith point-to-grid coordinates, when coupled with outputs from a digitalcompass, inclinometer and GPS receiver.

In one embodiment of the multi-coincident view camera, there may beseparate, steerable, co-incident fields of view, such as 30°, 10° and1°, with automated POI or multiple POIs tracking, face and irisrecognition, onboard matching and communication wirelessly over 256-bitAES encrypted UWB with laptop, combat radio, or other networked ormesh-networked device. The camera may network to CP's, TOC's andbiometric databases and may include a 3-axis, gyro-stabilized, highdynamic range, high resolution sensor to deliver the ability to see inconditions from a glaring sun to extremely low light. IDs may be madeimmediately and stored and analyzed locally or in remote storage. Thecamera may feature “look and locate” accurate geo-location of POI's andthreats, to >1,000 m distance, integrated 1550 nm, eye-safe laser rangefinder, networked GPS, 3-axis gyro, 3-axis magnetometer, accelerometerand inclinometer, electronic image enhancement and augmenting electronicstabilization aids in tracking, recording full-motion (30 fps) colorvideo, be ABIS, EBTS, EFTS and JPEG 2000 compatible, and meet MIL-STD810 for operation in environmental extremes. The camera may be mountedvia a gimbaled ball system that integrates mobile uncooperativebiometric collection and identification for a stand off biometriccapture solution as well as laser range-finding and POI geo-location,such as at chokepoints, checkpoints, and facilities. Multi-modalbiometric recognition includes collecting and identifying faces andirises and recording video, gait and other distinguishing marks ormovements. The camera may include the capability to geo-location tag allPOI's and collected data with time, date and location. The camerafacilitates rapid dissemination of situational awareness tonetwork-enabled units CP's and TOC's.

In another embodiment of the multi-coincident view camera, the camerafeatures 3 separate, Color VGA SWIR Electro-optic Modules that provideco-incident 20°, 7.5° and 2.5° Fields of View and 1 LWIR ThermalElectro-optic Modules for broad area to pinpoint imaging of POIs andTargets in an ultra-compact configuration. The 3-axis, gyro-stabilized,high dynamic range, color VGA SWIR cameras deliver the ability to see inconditions from a glaring sun to extremely low light as well as throughfog, smoke and haze—with no “blooming Geo-location is obtained byintegration of Micro-Electro-Mechanical System (MEMS) 3-axis gyroscopesand 3-axis accelerometers which augment the GPS receiver andmagnetometer data. Integrated 1840 nm, eye-safe laser range finder andtarget designator, GPS receiver and IMU provide “look and locate”,accurate geo-location of POIs and threats, to a 3 km distance. Thecamera displays and stores full-motion (30 fps) color video in its“camcorder on chip”, and stores it on solid state, removable drives, forremote access during flight or for post-op review. Electronic imageenhancement and augmenting electronic stabilization aids in tracking,geo-location range-finding and designation of POIs and targets. Thus,the eyepiece 100 delivers unimpeded “sight” of the threat by displayingthe feed from the multi-coincident view camera. In certain embodimentsof the eyepiece 100, the eyepiece 100 may also provide an unimpeded viewof the soldier's own weapon with “see through”, flip up/down,electro-optic display mechanism showing sensor imagery, moving maps, anddata. In one embodiment, the flip up/down, electro-optic displaymechanism may snap into any standard, MICH or PRO-TECH helmet's NVGmount.

FIG. 77 depicts an embodiment of a multi-coincident view camera,including laser range finder and designator 7702, total internalreflecting lens 7704, mounting ring 7708, total internal reflecting lens7710, total internal reflecting lens 7714, anti-reflection honeycombring 7718, 1280×1024 SWIR 380-1600 nm sensor 7720, anti-reflectionhoneycomb ring 7722, 1280×1024 SWIR 380-1600 nm sensor 7724,anti-reflection honeycomb ring 7728, and 1280×1024 SWIR 380-1600 nmsensor 7730. Other embodiments may include additional TIR lenses, a FLIRsensor, and the like.

Referring to FIG. 78, a flight eye is depicted. The feed from the flighteye may be transmitted to the eyepiece 100 or any other suitable displaydevice. The flight eye may include multiple individual SWIR sensorsmounted in a folded imager array with multiple FOVs. The flight eye is alow profile, surveillance and target designating system that enables acontinuous image of a whole battlefield in a single flyover, with eachsensor at VGA to SXGA resolution, day or night, through fog, smoke andhaze. Its modular design allows selective, fixed resolution changes inany element from 1° to 30° for telephoto to wide angle imaging in anyarea of the array. Each SWIR imager's resolution is 1280×1024 andsensitive from 380-1600 nm. A multi-DSP array board “stitches” all theimagery together and auto-subtracts the overlapping pixels for aseamless image. A coincident 1064 nm laser designator and rangefinder7802 can be mounted coincident with any imager, without blocking itsFOV.

Referring to FIG. 106, the eyepiece 100 may operate in conjunction withsoftware internal applications 7214 for the eyepiece that may bedeveloped in association with an eyepiece application developmentenvironment 10604, where the eyepiece 100 may include a projectionfacility suitable to project an image onto a see-through or translucentlens, enabling the wearer of the eyepiece to view the surroundingenvironment as well as the displayed image as provided through thesoftware internal application 7214. A processor, which may include amemory and an operating system (OS) 10624, may host the softwareinternal application 7214, control interfaces between eyepiece command &control and the software application, control the projection facility,and the like.

In embodiments, the eyepiece 100 may include an operating system 10624running on a multimedia computing facility 7212 that hosts an softwareinternal application 7214, wherein the internal application 7214 may bea software application that has been developed by a third-party 7242 andprovided for download to the eyepiece 100, such as from an app store10602, a 3D AR eyepiece app store 10610, from third party networkedapplication servers 10612, and the like. The internal application 7214may interact with the eyepiece control process facility 10634 processes,such as in conjunction with an API 10608, through input devices 7204,external devices 7240, external computing facilities 7232, command andcontrol 10630 facilities of the eyepiece, and the like. Internalapplications 7214 may be made available to the eyepiece 100 through anetwork communications connection 10622, such as the Internet, a localarea network (LAN), a mesh network with other eyepieces or mobiledevices, a satellite communications link, a cellular network, and thelike. Internal applications 7214 may be purchased though an applicationsstore, such as the app store 10602, 3D AR eyepiece app store 10610, andthe like. Internal applications 7214 may be provided through a 3D AReyepiece store 10610, such as software internal applications 7214specifically developed for the eyepiece 100.

An eyepiece applications development environment 10604 may be availablefor software developers to create new eyepiece applications (e.g. 3Dapplications), modify base applications to create new 3D applicationversions of the base application, and the like. The eyepiece applicationdevelopment environment 10604 may include a 3D application environmentthat is adapted to provide a developer with access to control schemes,UI parameters and other specifications available on the eyepiece oncethe finished application is loaded on or otherwise made functional forthe eyepiece. The eyepiece may include API 10608 that is designed tofacilitate communications between the finished application and theeyepiece computing systems. The application developer, within thedeveloper's development environment may then focus on developing anapplication with certain functionality without concerning themselveswith particulars of how to interact with the eyepiece hardware. The APImay also make it more straightforward for a developer to modify anexisting application to create a 3D application for use on the eyepiece100. In embodiments, an internal application 7214 may utilize networkedservers 10612 for client-server configurations, hybrid client-serverconfigurations (e.g. running the internal application 7214 in partlocally on the eyepiece 100 and in part on the application server 7214),hosting the application completely on the server, downloaded from theserver, and the like. Network data storage 10614 may be provided inassociation with the internal application 7214, such as in furtherassociation with application servers 10612, purchased applications, andthe like. In embodiments, internal applications 7214 may interact with asponsor facility 10614, markets 10620, and the like, such as to providesponsored advertisements in conjunction with the execution of theinternal application 7214, to provide marketplace content to the user ofthe eyepiece 100, and the like.

Referring to FIG. 107, the eyepiece application development environment10604 may be used for development of applications that may be presentedto the app store 10602, the 3D AR eyepiece app store 10610, and thelike. The eyepiece application development environment 10604 may includea user interface 10702, access to control schemes 10704, and the like.For instance, a developer may utilize menus and dialog boxes within theuser interface for accessing control schemes 10704 for selection so theapplication developer may choose a scheme. The developer may be ableselect a template scheme that generally operates the application, butmay also have individual controls that may be selected for variousfunctions that may override the template function scheme at a point inthe application execution. The developer may also be able to utilize theuser interface 10702 to develop applications with control schemes with afield of view (FOV) control, such as through a FOV interface. The FOVinterface may provide a way to go between a FOV that shows both displays(for each eye) and a single display. In embodiments, 3D applications forthe eyepiece may be designed within the single display view because theAPI 10610 will provide the translation which determines which display tobe used for which content, although developers may be able to select aspecific eye display for certain content. In embodiments, developers maybe able to manually select and/or see what is going to be displayed ineach eye, such as through the user interface 10702.

The eyepiece may have a software stack 10800 as described in FIG. 108.The software stack 10800 may have a head-mounted hardware and softwareplatform layer 10818, an interface-API-wrapper to platform layer 10814,libraries for development 10812 layer, an applications layer 10801, andthe like. The applications layer 10801 may in turn include consumerapplications 10802, enterprise applications 10804, industrialapplications 10808, and other like applications 10810. In addition,hardware 10820 associated with the execution or development of internalapplications 7214 may also be incorporated into the software stack10800.

In embodiments, the user experience may be optimized by ensuring thatthe augmented images are in focus with respect to the surroundingenvironment and that the displays are set at the appropriate brightnessgiven the ambient light and the content being displayed.

In an embodiment, the eyepiece optical assembly may include anelectrooptic module, aka display, for each eye that delivers content ina stereoscopic manner. In certain cases, a stereoscopic view is notdesired. In embodiments, for certain content, only one display may beturned on or only one electrooptic module may be included in the opticalassembly. In other embodiments, the brightness of each display may bevaried so that the brain ignores the dimmer display. An auto-brightnesscontrol of the image source may control the brightness of the displayedcontent based on the brightness in the environment. The rate ofbrightness change may depend on the change in the environment. The rateof brightness change may be matched to the adaptation of the eye. Thedisplay content may be turned off for a period following a sudden changein environment brightness. The display content may be dimmed with adarkening of the environment. The display content may get brighter witha brightening of the environment.

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software, program codes,and/or instructions on a processor. The processor may be part of aserver, a cloud server, client, network infrastructure, mobile computingplatform, stationary computing platform, or other computing platform. Aprocessor may be any kind of computational or processing device capableof executing program instructions, codes, binary instructions and thelike. The processor may be or include a signal processor, digitalprocessor, embedded processor, microprocessor or any variant such as aco-processor (math co-processor, graphic co-processor, communicationco-processor and the like) and the like that may directly or indirectlyfacilitate execution of program code or program instructions storedthereon. In addition, the processor may enable execution of multipleprograms, threads, and codes. The threads may be executed simultaneouslyto enhance the performance of the processor and to facilitatesimultaneous operations of the application. By way of implementation,methods, program codes, program instructions and the like describedherein may be implemented in one or more thread. The thread may spawnother threads that may have assigned priorities associated with them;the processor may execute these threads based on priority or any otherorder based on instructions provided in the program code. The processormay include memory that stores methods, codes, instructions and programsas described herein and elsewhere. The processor may access a storagemedium through an interface that may store methods, codes, andinstructions as described herein and elsewhere. The storage mediumassociated with the processor for storing methods, programs, codes,program instructions or other type of instructions capable of beingexecuted by the computing or processing device may include but may notbe limited to one or more of a CD-ROM, DVD, memory, hard disk, flashdrive, RAM, ROM, cache and the like.

A processor may include one or more cores that may enhance speed andperformance of a multiprocessor. In embodiments, the process may be adual core processor, quad core processors, other chip-levelmultiprocessor and the like that combine two or more independent cores(called a die).

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software on a server,client, firewall, gateway, hub, router, or other such computer and/ornetworking hardware. The software program may be associated with aserver that may include a file server, print server, domain server,internet server, intranet server and other variants such as secondaryserver, host server, distributed server and the like. The server mayinclude one or more of memories, processors, computer readable media,storage media, ports (physical and virtual), communication devices, andinterfaces capable of accessing other servers, clients, machines, anddevices through a wired or a wireless medium, and the like. The methods,programs or codes as described herein and elsewhere may be executed bythe server. In addition, other devices required for execution of methodsas described in this application may be considered as a part of theinfrastructure associated with the server.

The server may provide an interface to other devices including, withoutlimitation, clients, other servers, printers, database servers, printservers, file servers, communication servers, distributed servers,social networks, and the like. Additionally, this coupling and/orconnection may facilitate remote execution of program across thenetwork. The networking of some or all of these devices may facilitateparallel processing of a program or method at one or more location. Inaddition, any of the devices attached to the server through an interfacemay include at least one storage medium capable of storing methods,programs, code and/or instructions. A central repository may provideprogram instructions to be executed on different devices. In thisimplementation, the remote repository may act as a storage medium forprogram code, instructions, and programs.

The software program may be associated with a client that may include afile client, print client, domain client, internet client, intranetclient and other variants such as secondary client, host client,distributed client and the like. The client may include one or more ofmemories, processors, computer readable media, storage media, ports(physical and virtual), communication devices, and interfaces capable ofaccessing other clients, servers, machines, and devices through a wiredor a wireless medium, and the like. The methods, programs or codes asdescribed herein and elsewhere may be executed by the client. Inaddition, other devices required for execution of methods as describedin this application may be considered as a part of the infrastructureassociated with the client.

The client may provide an interface to other devices including, withoutlimitation, servers, other clients, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location. In addition, any of the devicesattached to the client through an interface may include at least onestorage medium capable of storing methods, programs, applications, codeand/or instructions. A central repository may provide programinstructions to be executed on different devices. In thisimplementation, the remote repository may act as a storage medium forprogram code, instructions, and programs.

The methods and systems described herein may be deployed in part or inwhole through network infrastructures. The network infrastructure mayinclude elements such as computing devices, servers, routers, hubs,firewalls, clients, personal computers, communication devices, routingdevices and other active and passive devices, modules and/or componentsas known in the art. The computing and/or non-computing device(s)associated with the network infrastructure may include, apart from othercomponents, a storage medium such as flash memory, buffer, stack, RAM,ROM and the like. The processes, methods, program codes, instructionsdescribed herein and elsewhere may be executed by one or more of thenetwork infrastructural elements.

The methods, program codes, and instructions described herein andelsewhere may be implemented on a cellular network having multiplecells. The cellular network may either be frequency division multipleaccess (FDMA) network or code division multiple access (CDMA) network.The cellular network may include mobile devices, cell sites, basestations, repeaters, antennas, towers, and the like. The cell networkmay be a GSM, GPRS, 3G, EVDO, mesh, or other networks types.

The methods, programs codes, and instructions described herein andelsewhere may be implemented on or through mobile devices. The mobiledevices may include navigation devices, cell phones, mobile phones,mobile personal digital assistants, laptops, palmtops, netbooks, pagers,electronic books readers, music players and the like. These devices mayinclude, apart from other components, a storage medium such as a flashmemory, buffer, RAM, ROM and one or more computing devices. Thecomputing devices associated with mobile devices may be enabled toexecute program codes, methods, and instructions stored thereon.Alternatively, the mobile devices may be configured to executeinstructions in collaboration with other devices. The mobile devices maycommunicate with base stations interfaced with servers and configured toexecute program codes. The mobile devices may communicate on a peer topeer network, mesh network, or other communications network. The programcode may be stored on the storage medium associated with the server andexecuted by a computing device embedded within the server. The basestation may include a computing device and a storage medium. The storagedevice may store program codes and instructions executed by thecomputing devices associated with the base station.

The computer software, program codes, and/or instructions may be storedand/or accessed on machine readable media that may include: computercomponents, devices, and recording media that retain digital data usedfor computing for some interval of time; semiconductor storage known asrandom access memory (RAM); mass storage typically for more permanentstorage, such as optical discs, forms of magnetic storage like harddisks, tapes, drums, cards and other types; processor registers, cachememory, volatile memory, non-volatile memory; optical storage such asCD, DVD; removable media such as flash memory (e.g. USB sticks or keys),floppy disks, magnetic tape, paper tape, punch cards, standalone RAMdisks, Zip drives, removable mass storage, off-line, and the like; othercomputer memory such as dynamic memory, static memory, read/writestorage, mutable storage, read only, random access, sequential access,location addressable, file addressable, content addressable, networkattached storage, storage area network, bar codes, magnetic ink, and thelike.

The methods and systems described herein may transform physical and/oror intangible items from one state to another. The methods and systemsdescribed herein may also transform data representing physical and/orintangible items from one state to another.

The elements described and depicted herein, including in flow charts andblock diagrams throughout the figures, imply logical boundaries betweenthe elements. However, according to software or hardware engineeringpractices, the depicted elements and the functions thereof may beimplemented on machines through computer executable media having aprocessor capable of executing program instructions stored thereon as amonolithic software structure, as standalone software modules, or asmodules that employ external routines, code, services, and so forth, orany combination of these, and all such implementations may be within thescope of the present disclosure. Examples of such machines may include,but may not be limited to, personal digital assistants, laptops,personal computers, mobile phones, other handheld computing devices,medical equipment, wired or wireless communication devices, transducers,chips, calculators, satellites, tablet PCs, electronic books, gadgets,electronic devices, devices having artificial intelligence, computingdevices, networking equipments, servers, routers, processor-embeddedeyewear and the like. Furthermore, the elements depicted in the flowchart and block diagrams or any other logical component may beimplemented on a machine capable of executing program instructions.Thus, while the foregoing drawings and descriptions set forth functionalaspects of the disclosed systems, no particular arrangement of softwarefor implementing these functional aspects should be inferred from thesedescriptions unless explicitly stated or otherwise clear from thecontext. Similarly, it will be appreciated that the various stepsidentified and described above may be varied, and that the order ofsteps may be adapted to particular applications of the techniquesdisclosed herein. All such variations and modifications are intended tofall within the scope of this disclosure. As such, the depiction and/ordescription of an order for various steps should not be understood torequire a particular order of execution for those steps, unless requiredby a particular application, or explicitly stated or otherwise clearfrom the context.

The methods and/or processes described above, and steps thereof, may berealized in hardware, software or any combination of hardware andsoftware suitable for a particular application. The hardware may includea general purpose computer and/or dedicated computing device or specificcomputing device or particular aspect or component of a specificcomputing device. The processes may be realized in one or moremicroprocessors, microcontrollers, embedded microcontrollers,programmable digital signal processors or other programmable device,along with internal and/or external memory. The processes may also, orinstead, be embodied in an application specific integrated circuit, aprogrammable gate array, programmable array logic, or any other deviceor combination of devices that may be configured to process electronicsignals. It will further be appreciated that one or more of theprocesses may be realized as a computer executable code capable of beingexecuted on a machine readable medium.

The computer executable code may be created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software, or any other machinecapable of executing program instructions.

Thus, in one aspect, each method described above and combinationsthereof may be embodied in computer executable code that, when executingon one or more computing devices, performs the steps thereof. In anotheraspect, the methods may be embodied in systems that perform the stepsthereof, and may be distributed across devices in a number of ways, orall of the functionality may be integrated into a dedicated, standalonedevice or other hardware. In another aspect, the means for performingthe steps associated with the processes described above may include anyof the hardware and/or software described above. All such permutationsand combinations are intended to fall within the scope of the presentdisclosure.

In embodiments, the augmented reality eyepiece (AR) of the presentinvention, e.g., AR eyepiece 100 in FIG. 1, is adapted to determineand/or compensate for the vergence of the user's eyes. Vergence is thesimultaneous rotation of the user's eyes around a vertical axis to movetheir respective optical axes in opposite directions to obtain ormaintain binocular vision. When a person looks at a closer object, theperson's eyes move their respective optical axes inwardly toward thenose, a composite motion that is known as convergence. To look at afarther object, the person's eyes move their respective optical axesoutwardly away from the nose, a composite motion that is known asdivergence. The person's eyes diverge until their respective opticalaxes are essentially parallel to each other when the person is fixatingon a point at infinity or very far away. Vergence works in conjunctionwith eye accommodation to permit a person to maintain a clear image ofan object as the object moves relative to the person. Vergencecompensation becomes important in situations where a virtual image,i.e., an AR image, such as a label or other information, is to be placednear to or overlap a real image or when a virtual image of an object isto be superimposed upon a real image of the object in order to make theplacement of the virtual image correct with respect to the real image.Methods of the present invention for vergence compensation and/ordetermination are described in the following paragraphs [00829] through[00850] and are collectively referred to as vergence methods.

The vergence methods may include a determination of the distance anobject of interest from the user of the AR eyepiece and subsequent useof that distance to determine the vergence angle, i.e, the angle formedby the intersection of the optical axes of the user's eyes as they lookat the object. The vergence angle is then used to determine the correctplacement of the AR image with respect to the object, which may be infront of, behind, or at the matched to the object. For example, in afirst set of vergence method embodiments, a single autofocus digitalcamera having a output signal is mounted in the AR eyepiece at someconvenient location, e.g., in the bridge section or near one of thetemples. The output of the camera is provided to a microprocessor withinthe AR eyepiece and/or transmitted to a remote processor. In eithercase, its signals relating to its autofocus capabilities are used todetermine the distance to objects which the user may see when the useris looking straight ahead. This distance, along with the interpupilarydistance of the user's eyes, is to determine the vergence and thecorrect placement of a virtual image, e.g., a label, which may bedesired for those objects. The distance and/or the vergence angle mayalso be used to determine the level of focus of the virtual object is tohave to be properly observable by the user. Optionally, additionalinformation about that particular user's vergence characteristics may beinput and stored in memory associated with the microprocessor and usedto adjust the determination of the vergence.

In a second set of vergence method embodiments, an electronic rangefinder that is independent of a camera is incorporated into the AReyepiece at some convenient location, e.g., in the bridge section ornear one of the temples. In these embodiments, the output of theelectronic range finder is used in the same manner as was the output ofthe autofocus camera described with regard to the first set of vergencemethod embodiments.

In a third set of vergence method embodiments, the AR eyepiece includesa plurality of range finding devices which may be autofocus camerasand/or electronic range finders. All of the plurality of devices may bealigned so as to determine the distance of objects in the same directionor one or more of the devices may be aligned differently from the otherdevices so that information about the distance to a variety of objectsis obtainable. The outputs from one or more of the devices are input andanalyzed in the same manner as was the output of the autofocus cameradescribed with regard to the first set of vergence methods.

In a fourth set of vergence method embodiments, one or more rangefinding devices are employed in the manner discussed above.Additionally, the AR eyepiece includes one or more eye-tracking deviceswhich are configured to track the movement and/or viewing direction ofone or both of the user's eyes. The output of the eye-tracking devicesis provided to a microprocessor within the AR eyepiece or may betransmitted to a remote processor. This output is used to determine thedirection the user is viewing, and, when eye-tracking information fromboth eyes is available, to determine the vergence of the user's eyes.This direction and, if available, vergence information is then usedalone or in conjunction with the vergence information determined fromthe range finding devices to determine placement, and optionally, thelevel of focus, of one or more virtual images related to one or moreobjects which the user may be viewing.

In a fifth set of vergence methods, one or more range finding device isdirected away from the direction that is straight ahead of the user ofthe AR eyepiece. Distances to objects detected by the range finderdevice are used to display virtual images of the object in the mannerdescribed above. Although the user may or may not be aware of thevirtual images when he is looking straight ahead, the user will be awareof the virtual images when the user looks in the direction of theobjects to which they are related.

A calibration sequence may be used with any of the vergence methodembodiments. The calibration sequence may employ steps of mechanicalcalibration nature, of an electronic calibration nature, or both. Duringthe calibration sequence, the interpupilarly distance of the user may bedetermined. Also, the user may be requested to look at a series of realor virtual objects having range of real or virtual distances, e.g., fromnear to far, and the vergence of the eyes is measured eithermechanically or electronically or both. The information from thiscalibration sequence may then be employed in the determinations ofvergence, focusing, and/or virtual image placement when the AR eyepiecesare in use. The calibration sequence is preferably employed when a userfirst puts on the AR eyepiece, but may be employed anytime the userbelieves that a recalibration would be helpful. Information correlatingthe user to that obtained during a calibration sequence may be storedfor use whenever that particular user identifies himself to the AReyepiece as its user, e.g., using any of the techniques described inthis document.

It is to be noted that some range finding devices use range determiningmethodologies in which information received from a device's sensors ismapped upon a space-representing rectilinear or non-rectilinear grid.The information from the various sectors of the grid is inter-comparedto determine the range distance. In the vergence method embodiments, theraw sensor information, the mapping information, the calculateddistance, or any combination of these may be used in the determinationof the placement and/or focus of the virtual image or images.

It is to be understood that the vergence method embodiments include theplacement of a virtual image for one of the user's eyes or for both ofthe user's eyes. In some embodiments, one virtual image is provided tothe user's left eye and a different virtual image is provided to theuser's right eye. In cases where multiple images are placed before theuser, whether or not the images are the same or different, the placementmay be simultaneous, at different times, or interlaced in time, e.g.,the images are shown at a predetermined flicker rate or rates (e.g., 30,60, and/or 180 Hz) with the image for the left eye being present whenthe image for the right eye is not and vice versa. In some embodiments,a virtual image is shown only to the person's dominant eye and in othersa virtual image is shown only to the person's non-dominant eye. In someembodiments which employ images which are interlaced in time, virtualimages of various objects which are located at various distances fromthe user are displayed in the manner described above; when the userlooks from the real image of one object to the real image of anotherobject, only the virtual image corresponding to the real image of theobject being viewed will be seen by the user's brain.

In embodiments, the invention provides methods for providing a depth cuewith augmented reality virtual objects or virtual information that canconvey a wide range of perceived depth to a broad range of individualswith different eye characteristics. These depth cue method embodimentsof the present invention use differences in the lateral positioning ordisparity of the augmented reality images provided to the two eyes ofthe individual to provide differences in the vergence of the virtualobjects or virtual information that convey a sense of depth. Oneadvantage of these methods is that the lateral shifting of the augmentedreality images can be different for different portions of the augmentedreality images so that the perceived depth is different for thoseportions. In addition, the lateral shifting can be done through imageprocessing of the portions of the augmented reality images. The user canexperience a full range of perceived depth through this method from asnear as the individual can focus to infinity regardless of theindividual's age.

In order to better understand these depth cue method embodiments of thepresent invention, it is useful to keep in mind that in some aspects ofaugmented reality, head mounted displays are used to add images ofvirtual objects or virtual information that are associated with the viewof a scene as seen by a user. To add additional effects to theperception of the augmented it is useful to place the virtual objects orvirtual information at a perceived depth in the scene. As an example, avirtual label can be placed onto an object in a scene such as the nameof a building. The perceived association of the virtual label with thebuilding is enhanced if the label and the building are perceived by theuser to be at the same depth in the scene. Head mounted displays withsee-through capabilities are well suited to providing augmented realityinformation such as labels and objects because they provide the userwith a clear view of the environment. However, for the augmented realityinformation to be of value, it must be easily associated with theobjects in the environment and as such the positioning of the augmentedreality information relative to the objects in the see-through view isimportant. While horizontal and vertical positioning of augmentedreality information is relatively straight forward if the head mounteddisplay has a camera that can be calibrated to the see-through view, thedepth positioning is more complicated. U.S. Pat. No. 6,690,393 describesa method for positioning 2D labels in a 3D virtual world. However, thismethod is not directed at displays with a see-through view where themajority of the image the user sees is not provided digitally and assuch the 3D location of objects is not known. U.S. Pat. No. 7,907,166describes a robotic surgical system using a stereo viewer in whichtelestration graphics are overlaid onto stereo images of an operatingsite. However, similar to the method described in U.S. Pat. No.6,690,393, this system uses captured images which are then manipulatedto add graphics and as such does not address the unique situation withsee-through displays wherein the majority of the image is not provideddigitally and the relative locations of objects that the user sees arenot known. Another prior art method for augmented reality is to adjustthe focus of the virtual objects or virtual information so that the userperceives differences in focus depth that provide a depth cue to theuser. As the user has to refocus his/her eyes to look at objects in thescene and to look at the virtual objects or virtual information, theuser perceives an associated depth. However, the range of depth that canbe associated with focus is limited by the accommodation that the user'seyes are capable of This accommodation can be limited in someindividuals particularly if the individual is older when eyes lose muchof their accommodation range. In addition, the accommodation range isdifferent depending on whether the user is near sighted or far sighted.These factors make the result of using focus cues unreliable for a largepopulation of user's with different ages and different eyecharacteristics. Therefore, the need persists beyond what is availablein the prior art for a widely useable method for associating depthinformation with augmented reality.

Some of the depth cue method embodiments of the present invention aredescribed in this and the following paragraphs [0XXX] through [0XYZ].Head mounted displays with see-through capabilities provide a clear viewof the scene in front of the user while also providing the ability todisplay an image, where the user sees a combined image comprised of thesee-through view with the display image overlaid. The methods are entailthe displaying of 3D labels and other 3D information using thesee-through display to aid the user in interpreting the environmentsurrounding the user. A stereo pair of images of the 3D labels and other3D information may be presented to the left and right eyes of the userto position the 3D labels and other 3D information at different depthsin the scene as perceived by the user. In this way the 3D labels andother 3D information can be more easily associated with the see-throughview and the surrounding environment.

FIG. 109 is an illustration of a head mounted display device 109100 withsee-through capabilities and is a special version of augmented realityeyepiece 100 shown in FIG. 1 and described throughout this document. Thehead mounted display device 109100 includes see-through displays 109110,stereo cameras 109120, electronics 109130, and range finder 109140. ThisWherein the electronics can include one or more of the following: aprocessor, a battery, a global positioning sensor (GPS), a directionsensor, data storage, a wireless communication system and a userinterface.

FIG. 110 is an illustration of the scene in front of the user as seen bythe user in the see-through view. A number of objects at differentdepths in the scene are shown for discussion. In FIG. 111 several of theobjects in the scene have been identified and labeled. However, thelabels are presented in two dimensional (2D) fashion either bypresenting the labels only to one eye of the user or by presenting thelabels at the same positions in the image to each eye so the labels arecoincident when viewed simultaneously. This type of labeling makes itmore difficult to associate the labels with the objects particularlywhen there are foreground and background objects as the labels appear tobe all located at the same perceived depth.

To make it easier to associate labels or other information with thedesired objects or aspects of the environment, it is advantageous topresent the labels or other information as three dimensional (3D) labelsor other 3D information so that the information is perceived by the userto be at different depths. This may be done by presenting 3D labels orother 3D information in overlaid images to the two eyes of the user witha lateral shift in position between the images that are overlaid ontothe see-through image so that the overlaid images have a perceiveddepth. This lateral shifting between images is also known as disparityto those skilled in stereo imaging and it causes the user to change therelative pointing of his/her eyes to align the images visually and thisinduces a perception of depth. The images with disparity are images ofthe 3D labels or other 3D information that are overlaid onto thesee-through view of the scene seen by the user. By providing 3D labelswith a large disparity, the user must align the optical axes of his/hereyes somewhat to bring the labels in the stereo images into alignmentwhich gives a perception of the labels being located close to the user.The 3D labels that have a small disparity (or no disparity) can bevisually aligned with the user's eyes looking straight ahead and thisgives the perception of the 3D labels being located at a distance.

FIGS. 112 and 113 illustrate a stereo image pair for 3D labels to beapplied to the see-through view shown in FIG. 110. FIG. 112 is an imageof the 3D labels shown to the user's left eye, while FIG. 113 is animage of the 3D labels shown to the user's right eye. Together, FIG. 112and FIG. 113 provide a stereo pair of images. In this stereo pair, thelateral positioning of the 3D labels is different between images shownin FIG. 112 and FIG. 113. FIG. 114 provides an overlaid image of FIG.112 and FIG. 113. For added clarity in FIG. 114, the 3D labels from FIG.113 have been shown in grey while the 3D labels from FIG. 112 are shownin black. In the foreground of FIG. 114, the 3D labels from FIG. 113 arepositioned to the left of the 3D labels from FIG. 112 with a relativelylarge disparity. In the background of FIG. 114, the 3D labels from FIG.113 are coincident with and positioned on top of the 3D labels from FIG.112 with no disparity. In the mid-ground region shown in FIG. 114, the3D labels from FIG. 112 and FIG. 113 have a medium disparity. Thisrelative disparity of the 3D labels as presented to the left and righteyes corresponds to the depth perceived by the user. By selecting adepth for the 3D labels that is coincident with the depth of the objectin the scene that the 3D label is associated with makes it easy for theuser to understand the connection between the 3D label and the object orother aspect of the environment that the user sees in the see-throughview. FIG. 115 shows the see-through view of the scene with the 3Dlabels showing their disparity. However, when viewed in real life, theuser would change the pointing direction of his/her eyes to make the 3Dlabels be coincident within each left/right set and it is this thatprovides the perception of depth to the user. The calculation ofdisparity is known to those skilled in the art. The equation forrelating disparity and distance is given by the equation

Z=Tf/d

where Z is the distance to the object from the stereo cameras, T is theseparation distance between the stereo cameras, f is the focal length ofthe camera lens, and d is the disparity distance on the camera sensorbetween images of the same object in the scene. Rearranging the term tosolve for the disparity, the equation becomes

d=TF/Z

For example, for 7 mm focal length cameras which are separated by 120 mmand used in conjunctions with image sensors having center-to-centerpixel distances of 2.2 micron, the disparities expressed in number ofpixels a visual target point is shifted when one display is compared tothe other are given Table 1 for some representative distances (given inmeters).

TABLE 1 Distance (m) Disparity (pixels) 1 381.8 2 190.9 10 38.2 50 7.6100 3.8 200 1.9

It is noted that sometimes in the art the disparity values for stereoimages are described using numbers which range from negative topositive, wherein zero disparity is defined for an object at a selecteddistance from the observer which the observer would perceive as being inthe mid-ground. The above-recited equations must be adapted to accountfor this shift of the zero point. When disparity values are described inthis way, the disparities of a close object and a far object may be thesame in magnitude and but opposite in sign.

FIG. 116 shows illustrations of the stereo pair of images captured bythe stereo cameras 109120 on the head mounted display device 109100.Since these images are captured from different perspectives, they willhave disparities that correspond to the distance from the head mounteddisplay device 109100. In FIG. 117, the two images from FIG. 116 areoverlaid to show the disparity between the images in the stereo pair.This disparity matches the disparity seen in the 3D labels shown for theobjects in FIGS. 114 and 115. As such, the 3D labels will be perceivedto be located at the same depth as the objects that they are intended tobe associated with FIG. 118 shows an illustration of the 3D labels asseen by the user as overlays to the see-through view seen with the leftand right eyes.

FIG. 119 is a flowchart for a depth cue method embodiment of the presentinvention. In step 119010, the electronics 109130 in the head mounteddisplay device 109100, determine the GPS location using the GPS for thehead mounted display device 109100. In step 119020, the electronics109130 determine the direction of view using an electronic compass. Thisenables the view location and view direction to be determined so thatobjects in the view and nearby objects can be located relative to theuser's field of view by comparing the GPS location of the head mounteddisplay device 109100 to databases of the GPS locations of other objectsin the head mounted display device 109100 or by connecting to otherdatabases using a wireless connection. In step 119030, objects ofinterest are identified relative to the user's field of view either bythe electronics 109130 analyzing databases stored on the device 109100or by wirelessly communicating in conjunction with another device. Instep 119040, distances to the objects of interest are determined bycomparing the GPS location of the head mounted display device 109100 tothe GPS locations of the objects of interest. Labels relating names orother information about the objects of interest are then generated alongwith disparities to provide 3D labels at distances perceived by the usercorresponding to the distances to the objects of interest in step119050. FIG. 111 shows examples of labels comprising names, distancesand descriptions for objects of interest in the user's field of view. Instep 119060, the 3D labels for the objects of interest are displayed tothe user's left and right eyes with the disparities to provide the 3Dlabels at the desired depths.

FIG. 120 is a flowchart for another a depth cue method embodiment of thepresent invention wherein steps similar to those in the steps of FIG.119 have been numbered using the same reference numerals as used in FIG.119. In step 120140, the distances and directions to the objects ofinterest relative to the user's field of view are determined either bythe electronics 109130 on the device or in conjunction with a wirelesslyconnected other device. In step 120160, 3D labels are displayed to theuser's left and right eyes with disparities to provide the 3D labels atthe desired depths, and in addition, the 3D labels are provided in theportions of the user's field of view that correspond to the direction tothe objects of interest. FIG. 111 shows an example where the label for adistant object of interest is provided toward the rear of the user'sfield of view and in the direction toward the distant object ofinterest, shown in this example as the label “10 miles to town thisdirection.” This feature provides a visual cue in the 3D informationwhich makes it easy for the user to navigate to objects of interest. Itshould be noted that the 3D labels can be provided in front of otherobjects in the see-through field of view.

FIG. 121 is a flowchart for yet another depth cue method embodiment ofthe present invention. In this embodiment, distances to objects ofinterest in the scene are determined with a distance measuring device109140 such as a rangefinder. In step 121010, one or more images arecaptured of a scene adjacent to the head mounted display device 109100using stereo cameras 109120. Alternately, a single camera may be used tocapture the one or more images of the scene. The one or more images ofthe scene can be different spectral types of images, for example, theimages can be visible-light images, ultraviolet images, infrared images,or hyperspectral images. The image or images are analyzed in step 121020to identify one or more objects of interest, wherein the analysis can beconducted by the electronics 109130 or the image can be sent wirelesslyto another device for analysis. In step 121030, distances to objects ofinterest are determined using the distance measuring device 109140.Disparities correlating correlate distances of the objects of interestare determined in step 121040. In step 121050, labels or otherinformation are determined for the objects of interest. In step 121060,the 3D labels or other 3D information are displayed for the objects ofinterest.

FIG. 122 is a flowchart for another depth cue method embodiment of thepresent invention. In this embodiment, the distances to objects in thescene are measured directly by using the stereo cameras to obtain adepth map of the scene. In step 122010, stereo cameras 109120 are usedto capture one or more stereo image sets of the scene adjacent to thehead mounted display device 109100. The one or more stereo image sets ofthe scene can be different spectral image types, for example, the stereoimages can be visible-light images, ultraviolet images, infrared images,or hyperspectral images. The stereo image set or sets are analyzed instep 122020 to identify one or more objects of interest, wherein theanalysis can be conducted by the electronics 109130 or the stereo imageset or sets can be sent wirelessly to another device for analysis. Instep 122030, the images in the stereo image set or sets are compared todetermine the disparities for the one or more objects of interest. Instep 122040, labels or other related to the one or more objects ofinterest are determined. In step 122050, 3D labels and/or 3D informationis displayed for the one or more objects of interest.

While the present disclosure includes many embodiments shown anddescribed in detail, various modifications and improvements thereon willbecome readily apparent to those skilled in the art. Accordingly, thespirit and scope of the present invention is not to be limited by theforegoing examples, but is to be understood in the broadest senseallowable by law.

All documents referenced herein are hereby incorporated by reference.

1. A system, comprising: an interactive head-mounted eyepiece worn by auser, wherein the eyepiece includes an optical assembly through whichthe user views a surrounding environment combined with displayedcontent, an integrated processor for handling content for display to theuser, and an integrated image source for introducing the content to theoptical assembly; and an auto-brightness control of the image sourcethat controls the brightness of the displayed content based on thebrightness in the environment.
 2. The system of claim 1, wherein therate of brightness change depends on the change in the environment. 3.The system of claim 1, wherein the rate of brightness change is matchedto the adaptation of the eye.
 4. The system of claim 1, wherein thedisplay content is turned off for a period following a sudden change inenvironment brightness.
 5. The system of claim 1, wherein the displaycontent is dimmed with a darkening of the environment.
 6. The system ofclaim 1, wherein the display content gets brighter with a brightening ofthe environment.